Two years ago I shared my experience on building the AWS Lambda function for a python project of my own. And a few days ago I stumbled upon a nice opensource CLI tool that I immediately wanted to transform in a web service.
Naturally, a simple, single-purpose tool is a perfect candidate for function-as-a-service (FaaS), and since I had past experience with AWS Lambda, this time I decided to meet its Google’s sibling - Google Cloud Function.
In this post we’ll discover how to take a python package with 3rd party dependencies, make a GCP Function from it and deploy it without a single click in the UI - all without leaving the IDE.
The python tool I considered a natural fit for a Cloud Function is a
pycatj by David Barroso that he released just recently.
For those with problems accessing yaml/json data: https://t.co/IbbS3x05bq— David Barroso (@dbarrosop) June 26, 2019
This tool helps you to map a JSON/YAML file to a Python dictionary highlighting the keys you need to access the nested data:
I felt like having a single-page web service that would do these transformations leveraging the
pycatj would be helpful to somebody sometime.
And as a starting point I decided to create a serverless function that will rely on
pycatj and will be triggered later by a Web frontend with an HTTP request carrying the content for pycatj-ifying.
In a nutshell, the function should behave something like that:
To add some sugar to the mix I will leverage the serverless framework to do the heavy lifting in a code-first way. The plan is set, lets go see it to completion.
The decomposition of the service creation and deployment can be done as follows:
- Google Cloud Platform
- Create a GCP account (if needed) and acquire the API credentials
- Create a project in GCP that will host a Function and enable the needed APIs for serverless to be able to create the Function and its artifacts
- Function creation and testing
- create the code in conformance with the GCP Function handlers/events rules
- Manage code dependencies
- Function deployment
- leveraging serverless framework to deploy a function to GCP
- Add a frontend (in another blog post) that will use the serverless function.
Following the agenda, ensure that you have a working GCP account (trial gives you $300, and GCP Function is perpetually FREE with the sane usage thresholds). Make sure that you have a billing account created, this is set up when you opt in the free trial program, for example. Without a linked billing account the Functions won’t work.
Once you have your account set, you should either continue with a default project or create a new one. In either case you need to enable the APIs that will be leveraged by serverless framework for a function deployment process. Go thru this guide carefully on how to enable the right APIs.
Do not forget to download your API credentials, as nothing can be done without them. This guide’s section explains it all.
The commands you will see in the rest of this post assume that the credentials are stored in
Since we are living on the edge, we will rely on the serverless framework to create & deploy our function. The very same framework I leveraged for the AWS Lambda creation, so why not try it for GCP Function?
The notable benefit of serverless framework is that it allows you to define your Function deployment as a code and thus making it repeatable, versionable and fast.
The way I start my serverless journey is by telling the serverless to generate a service template in the programming language of my choice. Later we can tune bits and pieces of that service, but if you start from a zero-ground, its easier to have a scaffolding to work on.
The result of the
serverless create --template <template> command will be a directory with a boilerplate code for our function and a few serverless artifacts.
We need to take a closer look at the generated
serverless.yml template file where we need to make some adjustments:
- the project name should match the project name you have created in the GCP
- the path to your GCP credentials json file should be valid
Given that the project in my GCP is called
pycatj and my credentials file is
provider section of the
serverless.yml file would look like this:
As to the
main.py generated by the framework, then its a simple boilerplate code with a text reply to an incoming HTTP request wrapped in a Flask object.
Lets test that our modifications work out so far by trying to deploy the template service.
Before we start pushing our function and its artifacts to the GCP, we need to tell serverless how to talk to the cloud provider. To do that, we need to install the
serverless-google-cloudfunctions plugin that is referenced in the
Install the google cloud functions plugin with the
npm install command using the directory with a generated serverless service files:
Note, here I mount my GCP credentials that are stored at
~/.gcloud dir to a containers
/root/.gcloud dir where serverless container will find them as they are referenced in the
And secondly I bind mount my project’s directory
~/projects/pycatj-web/pycatj-serverless to the
/opt/app dir inside the container that is a
WORKDIR of that container.
Now we have a green flag to try out the deployment process with
If the deployment fails with the Error Not Found make sure that you don’t have stale failed deployments by going to Cloud Console -> Deployment Manager and deleting all deployments created by Serverless
Upon a successful deployment you will have a Cloud Function deployed and reachable by the service URL:
curl-ing that API endpoint will return a simple “Hello world” as coded in our boilerplate
You can also verify the resources that were created by this deployment by visiting the Deployment Manager in the GCP console as well as navigating to the functions page and examine the deployed function and its properties:
Lets see how the the actual Python function is coupled to a service definition inside the serverless file. How about we give our function a different name by first changing the
functions section of the
Since we changed the function and the handler name to
pycatjify we should do the same to our function inside the
Deploying this function will give us a new API endpoint aligned to a new function name we specified in the
Up until now we played with a boilerplate code with a few names changed to give our function a bit of an identity. We reached the stage when its time to onboard the
pycatj package and make our function benefit from it.
Since the Functions are executed in the sandboxes on the cloud platforms, we must somehow tell what dependencies we want these sandbox to have when running our code. In the AWS Lambda example we packaged the 3rd party libraries along the function (aka vendoring).
In GCP case the vendoring approach is also possible and is done in the same way, but it is also possible to ship a pip
requirements.txt file along your
main.py that will specify your function dependencies as pythonistas used to.
Read more on GCP python dependency management
Unfortunately, the PIP version that GCP currently uses does not support PEP 517, so it was not possible to specify
-e git+https://github.com/dbarrosop/pycatj.git#egg=pycatj in a requirements file, thus I continued with a good old vendoring technique:
pycatj package and its dependencies in a
vendored directory and will be considered as Function’s artifact and pushed to GCP along the
main.py with the next
serverless deploy command.
Every function should be triggered by an event or a trigger that is supported by a cloud provider. When serverless is used the event type is specified for each function in the
With this configuration we expect our function to execute once an HTTP request hits the function API endpoint.
Yes, a thousand words later we finally at a milestone where we write actual python code for a function. The template we generated earlier gives us a good starting point - a function body with a single Flask
The logic of our serverless function that we are coding here is:
- parse the contents of an incoming HTTP request extracting the contents of a JSON file passed along with it
- transform the received data with
pycatjpackage and send back the response
With a few additions to access the
pycatj package in a
vendored directory and being able to test the function locally, the resulting
main.py file looks as follows:
This code has some extra additions to a simple two-step logic I mentioned before. I stuffed a default
data value that will be used when the incoming request has no body, then we will use this dummy data just for demonstration purposes.
To let me test the function code locally I added the
if __name__ == "__main__": condition and lastly I wrote some
Logging is a bless! Having a chance to look what happens with your function in a cloud platform sandbox is definitely a plus. With GCP the logging can be done in the simple and advanced modes. A simple logging logs everything that is printed by a function into
stdout/stderr outputs -> a simple
print() function would suffice. In a more advanced mode you would leverage a GCP Logging API.
The logs can be viewed with the Web UI Logging interface, as well as with the
gcloud CLI tool.
We previously already tried the deployed process with a boilerplate code just to make sure that the serverless framework works. Now that we have our
pycatj package and its dependencies stored in a
vendored folder and the function body is filled with the actual code, lets repeat the deployment and see what we get:
All goes well and serverless successfully updates our function to include the vendored artifacts as well as the new code in the
main.py. Under the hood, the deployment process took the code of our Function and packaged it into a directory, zipped and uploaded to the deployment bucket.
As demonstrated above, the serverless framework allows a user to express the deployment in a code, making the process extremely easy and fast.
Time to give our Function a roll by bombing it with HTTP requests. In this section I will show you how you can use the pycatjify service within a CLI and in a subsequent post we will write a simple Web UI using the API that our function provides.
With an empty GET request the function delivers a demo of its capabilities by taking a hardcoded demo JSON and making a transformation. The returned string is returned in a JSON object accessible by the
Getting a demo response back is useless, to make use of a pycatjify service a user can specify the
root value and pass the original JSON data in a POST request body using the
It is also allowed to omit the
root key, in that case a default root value will be applied:
My personal favorite is dumping a JSON file in a request. In that case a lengthy
curl is not needed and you can specify a path to a file with a
This example leverages the logic embedded in a function that treats the whole body of an incoming request as a data for
pycatj functionality available withing a HTTP call reach makes it possible to create a simple one-page web frontend that will receive the users input and render the result of the pycatj-web service we deployed in this post.
I will make another post covering the learning curve I needed to climb on to create a modern Material UI frontend that leverages the serverless function.