Cloud-native revolution pointed out the fact that the microservice is the new building block and your best friends now are Containers, AWS, GCE, Openshift, Kubernetes, you-name-it. But suddenly micro became not that granular enough and people started talking about serverless functions!
When I decided to step in the serverless property I chose AWS Lambda as my instrument of choice. As for experimental subject, I picked up one of my existing projects - a script that tracks new documentation releases for Nokia IP/SDN products (which I aggregate at nokdoc.github.io).
Given that not so many posts are going deeper than onboarding a simplest function, I decided to write down the key pieces I needed to uncover to push a real code to the Lambda.
Buckle up, our agenda is fascinating:
- testing basic Lambda onboarding process powered by Serverless framework
- accessing files in AWS S3 from within our Lambda with
boto3package and custom AWS IAM role
- packaging non-standard python modules for our Lambda
- exploring ways to provision shared code for Lambdas
- and using path variables to branch out the code in Lambda
What I am going to lambdsify is an existing python3 script called nokdoc-sentinel which has the following Lambda-related properties:
- uses non standard python package –
- reads/writes a file.
I specifically emphasized this non-std packages and relying on persistence since these aspects are not covered in 99% of Lambda-related posts, so, filling the spot.
AWS Lambda is a compute service that lets you run code without provisioning or managing servers. AWS Lambda executes your code only when needed and scales automatically, from a few requests per day to thousands per second.
Multiple choices are exposed to you when choosing an instrument to configure & deploy an AWS Lambda:
While it might be good to feel the taste of a manual Lambda configuration process through the AWS Console, I decided to go “everything as a code” way and use the Serverless framework to define, configure and deploy my first Lambda.
The Serverless Framework helps you develop and deploy your AWS Lambda functions, along with the AWS infrastructure resources they require. It’s a CLI that offers structure, automation and best practices out-of-the-box, allowing you to focus on building sophisticated, event-driven, serverless architectures, comprised of Functions and Events.
First things first, install the framework and configure AWS credentials. I already had credentials configured for AWS CLI thus skipped that part, if that is not the case for you, the docs are comprehensive and should have you perfectly covered.
Once serverless is installed, start with creating an
serviceis like a project. It’s where you define your AWS Lambda Functions, the events that trigger them and any AWS infrastructure resources they require, all in a file called serverless.yml.
Two files will be created:
handler.py– a module with Lambda function boilerplate code
serverless.yml– a service definition file
handler.py module to
sentinel.py, also changed the enclosed function' name and deleted redundant code from the template. For starters I kept the portion of a sample code just to test that deploying to AWS via serverless actually works.
Thing to remember is that you also must to make appropriate changes in the
serverless.yml, once you renamed the module and the function names:
Before adding some actual load to the Lambda function, lets test that the deployment works. To trigger Lambda execution I added HTTP GET event with the
test path in the
serverless.yml file. So a call to
https://some-aws-hostname.com/test should trigger our lambda function to execute.
Read all about supported by serverless framework events in the official docs.
And we are coming to the first test deployment with the following assets:
Lets go and deploy:
Note the endpoint URL at the bottom of the output, using this API endpoint we can check if our Lambda is working:
Serverless deployed the Lambda using some defaults parameters (region: us-east-1, stage: dev, IAM role); plus serverless did some serious heavy-lifting in order to deploy our code to AWS. In particular:
- archived the project files as a zip archive and loaded it to AWS S3
- created CloudFormation template that defines all the steps needed to onboard a Lambda and setup an API gateway to respond to
Key artifacts that were created by serverless in AWS can be browsed with the AWS CLI:
Are you interested what is in this archive
There are two files we dealt with earlier plus
.vscode dir that a text editor created for its settings. Having
.vscode in the deployment package actually indicates that by default serverless zipped everything in the project' dir. You can get in control of this process by using include/exclude statements.
It is natural that AWS assumes that Lambdas will be used in a close cooperation with the rest of the AWS family. And for the file storage AWS S3 is a one-stop shop.
What you have to sort out before digging into S3 interaction is the permissions that your Lambda has. When serverless deployed our Lambda with a lot of defaults it also handed out a default IAM role to our Lambda:
To be able to interact with AWS S3 object model, this Role should have access to S3. Lets investigate:
As you see S3 access is not a part of default permissions, so we must grant it to our Lambda. Instead of adding additional permissions to the existing role manually, we can re-deploy the Lambda with updated
serverless.yml file. In this edition I specified availability zone, set existing S3 bucket as a deployment target and included IAM role configuration allowing full-access to S3 objects:
Now the re-deployment will create another Lambda (hence the availability zone has changed), deploy the code in the existing bucket
rdodin and apply a policy that allows S3 interaction.
Now as the S3 permissions are there, we are free to list bucket contents and modify the files in it.
AWS provides us with the boto3 package as a Python API for AWS services. Moreover, this package comes pre-installed on the system that is used to run the Lambdas, so you do not need to provide a package.
I put a file (releases_current.json) that my script expects to read to the directory created by the serverless deployment script:
Lets see if we can access it from within the Lambda using
boto3 and its documentation:
Re-deploy and check:
So far, so good. We are now capable of reading/writing to a file stored in AWS S3.
We were lucky to use only the packages that either standard (
json) or comes preinstalled in Lambda-system (
boto3). But what if we need to use packages other from that, maybe your own packages or from PyPI?
Well, in that case you need to push these packages along with your function' code as a singe deployment package. As official guide says you need to copy packages to the root directory of your function and zip everything as a single archive.
What comes as a drawback of this recommendation is that
- your project dir will be dirty with all these packages sitting in the root
- you will have to .gitignore these packages directory to keep your packages out of a repository
I like the solution proposed in the “Building Python 3 Apps On The Serverless Framework” post. Install your packages in a some directory in your projects dir and modify your
PYTHONPATH to include this directory.
Now modify your code to include
vendored directory in your
Note, that if a package has a native binary code, it must be compiled for the system that is used to run Lambdas.
Even though a Lambda often assumed as an independent function, a real application you might want to transfer to Lambda quite likely will have dependencies on some common code. Refer to the “Writing Shared Code” section of the above mentioned blog post to see how its done.
Another common practice in a classic util function is to have some arguments (argparse) that allow to branch out the code and make an app' logic feature-rich. In Lambdas, of course, you have no CLI exposed, so to make a substitution for the arguments you can go two ways:
- create several functions for your project and bind different API endpoints to each of them
- use a single function and add a variable part to the API endpoint
I will show how to handle the latter option. First, create a variable parameter for your API endpoint in the
Now you Lambda can be branched out like that, using the part that you will place in the end of your API endpoint as an argument.
Now adding an arbitrary text after the
go/ path will be evaluated in your Lambda allowing you to conditionally execute some parts of your code.
With the above explained concepts I successfully transferred nokdoc-sentinel script from a standalone cron-triggered module to the AWS Lambda. You can check out the project' code and the
serverless.yml file at github repo.
- Benny Bauer – Python in The Serverless Era PyCon 2017
- Ryan S. Brown – Building Python 3 Apps On The Serverless Framework
- Serverless Framework AWS Guide
- AWS Lambda Developers Guide
- AWS CLI Command Reference
- Keeping secrets out of Git with Serverless
Post comments are here.