Building AWS Lambda with Python, S3 and serverless

July 24, 2017

Cloud-native revolution pointed out the fact that the microservice is the new building block and your best friends now are Containers, AWS, GCE, Openshift, Kubernetes, you-name-it. But suddenly micro became not that granular enough and people started talking about serverless functions!

Brian Christner, Docker & Serverless: https://www.slideshare.net/BrianChristner/docker-serverless

Brian Christner, Docker & Serverless: https://www.slideshare.net/BrianChristner/docker-serverless

When I decided to step in the serverless property I chose AWS Lambda as my instrument of choice. As for experimental subject, I picked up one of my existing projects - a script that tracks new documentation releases for Nokia IP/SDN products (which I aggregate at nokdoc.github.io).

Given that not so many posts are going deeper than onboarding a simplest function, I decided to write down the key pieces I needed to uncover to push a real code to the Lambda.

Buckle up, our agenda is fascinating:

testing basic Lambda onboarding process powered by Serverless framework
accessing files in AWS S3 from within our Lambda with boto3 package and custom AWS IAM role
packaging non-standard python modules for our Lambda
exploring ways to provision shared code for Lambdas
and using path variables to branch out the code in Lambda

Continue reading

Author's picture

Roman Dodin

Eagerness to learn & passion to share

Netdevops @ Nuage Networks

Russia

no post found

Saturating the network with FTP

Nov 11, 2018

While working on the Ipanema Wan Opt VNF integration with Nuage Networks I stumbled upon an interesting case which required to max out the network with FTP traffic. The tricky point there was to create the FTP connection which won’t be limited by the disk IO performance. Especially, considering that the disks were kind of slow in the setup I had.

It turns out, you can use the in-memory devices in the FTP communication path /dev/zero -> /dev/null, ruling out the slowliness that could have been added by the disks. Lets figure out how to do that!

Prepping up for and passing the Certified Openstack Administrator Exam

Nov 11, 2018

Shortly after I passed AWS CSA exam I went on a hunt for the next certification to claim. Decided to tackle the Openstack COA certification first saving the Docker/Kubernetes certification for a later occasion.

There is a common joke floating around: “Oh, is Openstack still a thing?” - yes, its pretty much still a thing, especially in the Telecom area where VMs are the only viable option for the most of the cases (think VNFs).
Openstack also powers our teams public SDN lab - NuageX - that allows to provision a fully functional Nuage environment in a matter of minutes. So I wanted to get a better operational knowledge of Openstack to be able to support and tune the platform if necessary.

Disclaimer: I took the recently updated COA exam which is based on the Openstack Pike release, although I did not face any Pike-specific questions during the exam. This does not mean that the exam content will stay the same throughout the course evolution, so watch out for the updates.

Prepping up for and passing the AWS Certified Solution Architect - Associate

May 5, 2018

On May 11th I passed the AWS Certified Solution Architect Associate exam which was harder then I expected. In this memo I will outline how I prepared to this exam, what topics you better pay more attention to and some tips and hints I could give to anyone going for AWS CSA exam.

Disclaimer: I took the original AWS CSA exam, not the one that was launched in Feb 2018; this older version is only available to schedule till August 2018. After that date the newer version of this exam will be the only one available. Watch out, it has a new set of objectives.

Uploading multiple files to AWS S3 in parallel

Mar 3, 2018

Have you ever tried to upload thousands of small/medium files to the AWS S3? If you had, you might also noticed ridiculously slow upload speeds when the upload was triggered through the AWS Management Console. Recently I tried to upload 4k html files and was immediately discouraged by the progress reported by the AWS Console upload manager. It was something close to the 0.5% per 10s. Clearly, the choke point was the network (as usual, brothers!).

Comer here, Google, we need to find a better way to handle this kind of an upload.

How to add YAML highlight in Highlight.js?

Nov 11, 2017

Haters gonna hate YAML, thats for sure. I am on the other hand in love with YAML; when one have to manually write/append config files I find YAML easier than JSON (and you have comments too). Ansible, various static-site-generators and quite a lot of opensource tools use YAML syntax for the configuration purposes. But still, YAML syntax highlighting is not a part of the Common languages shipped with highlight.

SROS Rootifier or how to flatten 7750 SR config

Nov 11, 2017

Back in the days when I mostly did routing stuff I spent the whole day configuring SROS devices via SSH. And once in a while I saw that SSH session or its server part (or even underlying connection) glitched, resulting in a corrupted lines feeded to the device.

What was also quite common is to make a mistake (i.e. syntax one) in a single line and watch like the rest of config got applied to the wrong context.

These sad facts pushed me to create a rootifier CLI script, that was converting tree-like SROS config into flattented (aka rooted) fashion.

rootifier

Now I decided to make a web service of that script, that is publicly available at https://rootifier.netdevops.me/

Flask application in a production-ready container

Nov 11, 2017

Flask documentation is very clear on where is the place for its built-in WSGI application server:

While lightweight and easy to use, Flask’s built-in server is not suitable for production as it doesn’t scale well and by default serves only one request at a time.

So how about I share with you a Dockerfile that will enable your Flask application to run properly and ready for production-like deployments? As a bonus, I will share my findings discovered along the way of building this container image.

nginx-uwsgi-flaks-alpine-docker

Waiting for SSH service to be ready with Paramiko

Oct 10, 2017

Today I faced a task which required first to establish an SSH tunnel in a background process and later use this tunnel for SSH connection. What seemed like a child’s play first actually had some fun inside.

A problem were hidden right between the moment you spawned ssh process in the background and the next moment you tried to use this tunnel. In other words, it takes literally no time to spawn a process in the background, but without checking that tunnel is ready, you will quite likely receive an error, since your next instructions will be executed immediately after.

Consequently, I needed a way to ensure that the SSH service is ready before I try to consume it.

scheme

Changing Libvirt bridge attachment in a running domain aka on-the-fly

Oct 10, 2017

At work I always prefer KVM hosts for reasons such as flexible, free and GUI-less. Yet I never bothered to go deeper into the networking features of Libvirt, so I only connect VMs to the host networks via Linux Bridges or OvS. Far far away from fancy virtual libvirt networks.

Even with this simple networking approach I recently faced a tedious task of reconnecting VMs to different bridges on-the-fly.
My use case came from a need to connect a single traffic generator VM to the different access ports of virtual CPEs. Essentially this meant that I need to reconnect my traffic generator interfaces to different bridges back and forth:

scheme

Apparently there is no such virsh command that will allow you to change bridge attachments for networking devices, so a bit of bash-ing came just handy.

Installing xrdp 0.9.1 on Ubuntu 16.04 Xenial

Aug 8, 2017

xrdp is defacto the default RDP server for Linux systems sharing with VNC the remote access solution olympus. I personally found it more resource friendly and feature rich compared to VNC solutions I tried.

The only problem I found with xrdp is that current Ubuntu LTS release Xenial 16.04 has a way outdated 0.6.1-2 version of xrdp in the packages repo. This version has no shared clipboard support, which makes remote support/remote access a tedious task.

xrdp currently in its 0.9.3 version and it would be really nice to have a more recent package, rather than installing it from sources, like many solutions propose.

Well, no need to compile xrdp from sources (unless you want to), because you can leverage a ppa from hermlnx that has xrdp 0.9.1-7 already built for amd64 and i386 systems

# all you need is
sudo add-apt-repository ppa:hermlnx/xrdp
sudo apt-get update
sudo apt-get install xrdp

You can also try a deb package of xrdp 0.9.2 – https://github.com/suminona/xrdp-ru-audio