Skip to content

didier-durand/lambda-cobol

Repository files navigation

Legacy Serverless Modernization: Cobol and AWS Lambda functions

Cobol Lambdas on AWS

Goal

This repository implements a fully automated Github Workflow to deploy and run as an AWS Lambda function a 'Hello World' Cobol program compiled with GnuCOBOL. GnuCOBOL compiles the source code as a native x86 module depending on the libcob library. Both are packaged and uploaded as a custom Lambda runtime by the workflow. The workflow is scheduled by cron on a minimalweekly basis to make sure that it keeps working.

The benefits of the serverless architecture are not reserved to newly written applications. The purpose of this showcase is to demonstrate how those benefits can be combined with legacy code, still "doing the job" and delivering solid business value, to further extend its life.

The deployed Cobol program is accessible over http via the definition of a REST service on the AWS API gateway. The results of the various executions of the workflow in this repo can be seen in Actions tab here above. Also, some highlights of last execution are reported in last section of this page.

This initial use case will be refined in upcoming versions by adding a database, calling subprograms, etc.

Lambda functions were chosen here because they are the canonical service proposed by AWS to support the strategic serverless architecture. Its virtues are described below. Lambdas can execute uninterruptedly at scale with no effort on the customer side.

The purpose of this unusual / unexpected use case with Cobol is to trigger further ideas around "serverless legacy". The Cobol numbers below will demonstrate that it makes quite a lot of sense to reuse the massive existing assets on a modern cloud platform to further extend their life and leverage them in new ways. It's especially attractive given the incredibly affordable costs of Lambdas when compared to costs of mainframe Mips!

Feel free to fork and replicate this repo in your own environment (see Setup section below). All feedback and suggestions for extensions welcome! (Please, open a Github issue ticket for this purpose). If you like this repository, please, give it a star!

Cobol

Cobol was initially specified more than 60 years ago by Grace Hopper, aka "the grandmother of Cobol". This language remains quite vivid despite its age. It is still heavily used in mainframe shops like banks, insurance companies, administrations, etc.

For example, issues at the beginning of the Covid-19 pandemic have shown how critical this programming language remains to run daily operations in US administrations. Old Cobol applications are still heavily used by several states to process unemployment claims.

A report of 2017 by Thomson Reuters states that 200+ billion lines of Cobol are still in operation. It also asserts that 43% of banking systems are built on Cobol and that 95% of ATM swipes rely on this language.

Cobol Use in Finance Service Industry (Thomson Reuters)

And this importance is not going to decline anytime soon: IBM reports that more than 5 billion additional lines are produced each year!

Lambda Functions and Serverless Architecture

Canonical Lambda architecture (Python example)

As per AWS documentation: "AWS Lambda is a serverless compute service that lets you run code without provisioning or managing servers, creating workload-aware cluster scaling logic, maintaining event integrations, or managing runtimes. With Lambda, you can run code for virtually any type of application or backend service - all with zero administration. Just upload your code as a ZIP file or container image, and Lambda automatically and precisely allocates compute execution power and runs your code based on the incoming request or event, for any scale of traffic."

It means that AWS Lambda service does the heavyweight lifting to the benefit of its users: all the non-functional requirements (NFRs) like high availability, scalability, security, resource and performance optimization, etc. are implemented by the AWS experts. Customer teams can remain focused on functional code and rely on this scalable platform to run their applications uninterruptedly at scale.

This serverless architecture is the optimal solution to run applications with very stringent SLAs (incl. extremely low RTO & RPO). Those highly demanding SLAs are usually out of reach of most organizations by themselves. With Lambdas, it become extremely easy as AWS encapsulates its domain expertise in the underlying platform design and makes the thorough investments to deploy and operate the underlying infrastructure in a multi-tenant fashion.

Additionally, the service strictly respects the "pay-per-use" model: AWS will charge only for the memory consumption and execution time of executed Lambdas. If they remain unused, no cost!

These virtues of serverless architecture make it the ideal target when the functional needs and technical requirements allow for it. It should be the primary goal of any corporate cloud architect to make maximum use of managed serverless features.

Serverless Application Model

AWS open sourced at end of 2016 the Serverless Application Model (SAM) framework to describe such serverless applications made of multiple Lamdba functions with all their dependencies. The main purpose of SAM is to reduce the effort by developers when creating such applications. Required artefacts and definitions are specified at a high-level of abstraction. The SAM processor on AWS, in collaboration with the CloudFormation service, which it extends, will take care of all the low-level definitions of corresponding required AWS resources to deploy the Lambda function and make it publicly accessible through the API gateway.

This allows the implementation of Infrastructure-as-Code best practices.

The SAM model for our Cobol Lambda is in the file lambda-cobol-sam.yaml. It contains a single AWS::Serverless::Function with its parameters. During deployment, SAM & CloudFormation expand it into 6 more granular resources that can be located with "Resource creation Initiated" in last section below.

Workflow and Components

Implemented as a Github Action, the workflow - scripted in lambda-cobol.sh - comprises following key steps:

  1. A Docker image is constructed to install the GnuCOBOL compiler and its dependencies on top of the base Amazon Linux image. The purpose of such a container is to leverage the isolation provided by containers. Consequently, the build environment is fully controlled.
  2. This Cobol builder imports the source code of hello-world.cob and compiles it to generate an x86 native binary named hello-world.
  3. This binary is packaged, via SAM CLI, with other required runtime artefacts. The libcob library is required by GnuCOBOL. The shell script bootstrap(name imposed by specifications) implements the requirements of custom Lambda runtimes.
  4. This package is deployed on the Lambda service via SAM CLI.
  5. The SAM description is processed by AWS Lambda and CloudFormation to deploy the function.
  6. SAM CLI is used to check proper deployment.
  7. SAM CLI invokes the function synchronously.
  8. curl calls the URL with the obtained DNS to validate the proper execution of the newly deployed Lambda. The URL for curl is built following template: https://$API_ID.execute-api.$AWS_REGION.amazonaws.com/Prod/$LAMBDA_NAME. $API_ID is dynamic and obtained via a aws apigateway get-rest-apis CLI command.

When a previous deployment of the CloudFormation stack is active, it gets deleted just before the SAM build to trigger the entire CloudFormation deployment process.

Note: the version of GnuCOBOL currently used is v2.2. A version 3.1 was published in late December, 2020. But, its libcob runtime library has hardwired dependencies on very recent Linux system libraries, that are not yet available with proper version in Lambda runtime. We'll bump to newest GnuCOBOL when Lambda runtime gets updated. The official repository for all available versions of GnuCOBOL is here.

Fork and Setup

When you fork this repository to run it on your own, you will need to recreate three Github secrets in your own repository for workflows to work properly:

  • ${{ secrets.AWS_ACCESS_KEY_ID }}: the access key under which the workflow will run
  • ${{ secrets.AWS_SECRET_ACCESS_KEY }}: the secret key validating the use of the above access key
  • ${{ secrets.AWS_REGION }}: the region in which you want the workflow to be deployed and executed

The credentials given to this identity via AWS IAM must grant permissions to deploy and run Lambda functions as well as create corresponding API gateway definitions. In addition, an S3 bucket must be created to import the uploaded artefacts in AWS when they get deployed.

Execution Highlights

Below are the logs of the last execution related to the Lamdba service operated from SAM CLI:

 
### execution date: Thu Mar  9 01:33:05 UTC 2023
 
### Check existing Lambdas functions...
{
    "Functions": [
        {
            "FunctionName": "Hello-world-Python",
            "FunctionArn": "arn:aws:lambda:us-east-1:514764745669:function:Hello-world-Python",
            "Runtime": "python3.8",
            "Role": "arn:aws:iam::514764745669:role/service-role/Hello-world-Python-role-lyqky200",
            "Handler": "lambda_function.lambda_handler",
            "CodeSize": 299,
            "Description": "",
            "Timeout": 3,
            "MemorySize": 128,
            "LastModified": "2021-02-06T10:48:38.267+0000",
            "CodeSha256": "fI06ZlRH/KN6Ra3twvdRllUYaxv182Tjx0qNWNlKIhI=",
            "Version": "$LATEST",
            "TracingConfig": {
                "Mode": "PassThrough"
            },
            "RevisionId": "d90d1b6d-667c-46d9-b9d5-e7fdefdfc004",
            "PackageType": "Zip",
            "Architectures": [
                "x86_64"
            ],
            "EphemeralStorage": {
                "Size": 512
            },
            "SnapStart": {
                "ApplyOn": "None",
                "OptimizationStatus": "Off"
            }
        }
    ]
}
 
### Starting SAM build...

Build Succeeded

Built Artifacts  : build
Built Template   : build/template.yaml

Commands you can use next
=========================
[*] Validate SAM template: sam validate
[*] Invoke Function: sam local invoke -t build/template.yaml
[*] Test Function in the Cloud: sam sync --stack-name {{stack-name}} --watch
[*] Deploy: sam deploy --guided --template-file build/template.yaml
 
### Starting SAM deployment...

	Deploying with following values
	===============================
	Stack name                   : lambda-cobol-stack
	Region                       : us-east-1
	Confirm changeset            : False
	Disable rollback             : False
	Deployment s3 bucket         : net.didier-durand.lambda-code
	Capabilities                 : ["CAPABILITY_IAM"]
	Parameter overrides          : {}
	Signing Profiles             : {}

Initiating deployment
=====================

Waiting for changeset to be created..
CloudFormation stack changeset
-------------------------------------------------------------------------------------------------
Operation                LogicalResourceId        ResourceType             Replacement            
-------------------------------------------------------------------------------------------------
+ Add                    HelloWorldCobolGetReso   AWS::Lambda::Permissio   N/A                    
                         urcePermissionProd       n                                               
+ Add                    HelloWorldCobolRole      AWS::IAM::Role           N/A                    
+ Add                    HelloWorldCobol          AWS::Lambda::Function    N/A                    
+ Add                    ServerlessRestApiDeplo   AWS::ApiGateway::Deplo   N/A                    
                         ymentaf1c952223          yment                                           
+ Add                    ServerlessRestApiProdS   AWS::ApiGateway::Stage   N/A                    
                         tage                                                                     
+ Add                    ServerlessRestApi        AWS::ApiGateway::RestA   N/A                    
                                                  pi                                              
-------------------------------------------------------------------------------------------------

Changeset created successfully. arn:aws:cloudformation:us-east-1:514764745669:changeSet/samcli-deploy1678325694/7f9ee7f0-fc45-4a76-a364-87bc073b74ec


2023-03-09 01:35:06 - Waiting for stack create/update to complete

CloudFormation events from stack operations (refresh every 0.5 seconds)
-------------------------------------------------------------------------------------------------
ResourceStatus           ResourceType             LogicalResourceId        ResourceStatusReason   
-------------------------------------------------------------------------------------------------
CREATE_IN_PROGRESS       AWS::IAM::Role           HelloWorldCobolRole      -                      
CREATE_IN_PROGRESS       AWS::IAM::Role           HelloWorldCobolRole      Resource creation      
                                                                           Initiated              
CREATE_COMPLETE          AWS::IAM::Role           HelloWorldCobolRole      -                      
CREATE_IN_PROGRESS       AWS::Lambda::Function    HelloWorldCobol          -                      
CREATE_IN_PROGRESS       AWS::Lambda::Function    HelloWorldCobol          Resource creation      
                                                                           Initiated              
CREATE_COMPLETE          AWS::Lambda::Function    HelloWorldCobol          -                      
CREATE_IN_PROGRESS       AWS::ApiGateway::RestA   ServerlessRestApi        -                      
                         pi                                                                       
CREATE_IN_PROGRESS       AWS::ApiGateway::RestA   ServerlessRestApi        Resource creation      
                         pi                                                Initiated              
CREATE_COMPLETE          AWS::ApiGateway::RestA   ServerlessRestApi        -                      
                         pi                                                                       
CREATE_IN_PROGRESS       AWS::ApiGateway::Deplo   ServerlessRestApiDeplo   -                      
                         yment                    ymentaf1c952223                                 
CREATE_IN_PROGRESS       AWS::Lambda::Permissio   HelloWorldCobolGetReso   -                      
                         n                        urcePermissionProd                              
CREATE_IN_PROGRESS       AWS::Lambda::Permissio   HelloWorldCobolGetReso   Resource creation      
                         n                        urcePermissionProd       Initiated              
CREATE_IN_PROGRESS       AWS::ApiGateway::Deplo   ServerlessRestApiDeplo   Resource creation      
                         yment                    ymentaf1c952223          Initiated              
CREATE_COMPLETE          AWS::ApiGateway::Deplo   ServerlessRestApiDeplo   -                      
                         yment                    ymentaf1c952223                                 
CREATE_IN_PROGRESS       AWS::ApiGateway::Stage   ServerlessRestApiProdS   -                      
                                                  tage                                            
CREATE_IN_PROGRESS       AWS::ApiGateway::Stage   ServerlessRestApiProdS   Resource creation      
                                                  tage                     Initiated              
CREATE_COMPLETE          AWS::ApiGateway::Stage   ServerlessRestApiProdS   -                      
                                                  tage                                            
CREATE_COMPLETE          AWS::Lambda::Permissio   HelloWorldCobolGetReso   -                      
                         n                        urcePermissionProd                              
CREATE_COMPLETE          AWS::CloudFormation::S   lambda-cobol-stack       -                      
                         tack                                                                     
-------------------------------------------------------------------------------------------------

Successfully created/updated stack - lambda-cobol-stack in us-east-1

 
 
### Inkoking deployed Lambda synchronously from CLI...
{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}
invocation result:

  {
    "isBase64Encoded": false,
    "statusCode": 200, 
    "body": "Hello World from COBOL!" 
  } 
 
### Obtaining API gateway config...
{
    "items": [
        {
            "id": "eioy58x8l3",
            "name": "lambda-cobol-stack",
            "createdDate": "2023-03-09T01:36:00+00:00",
            "version": "1.0",
            "apiKeySource": "HEADER",
            "endpointConfiguration": {
                "types": [
                    "EDGE"
                ]
            },
            "tags": {
                "aws:cloudformation:logical-id": "ServerlessRestApi",
                "aws:cloudformation:stack-id": "arn:aws:cloudformation:us-east-1:514764745669:stack/lambda-cobol-stack/9843f110-be1a-11ed-a5ac-0e778226314b",
                "aws:cloudformation:stack-name": "lambda-cobol-stack"
            },
            "disableExecuteApiEndpoint": false
        }
    ]
}
api id: eioy58x8l3
 
### Running curl https request to https://eioy58x8l3.execute-api.us-east-1.amazonaws.com/Prod/lambda-cobol-hello-world ...
Hello World from COBOL!