Running multiple VMs out of the same disk image is something we, network engineers, do quite often. A virtualized network usually consists of a few identical virtualized network elements that we interconnected with links making a topology.
In the example above we have 7 virtualized routers in total, although we used only two VM images to create this topology (virtualized Nokia router and it’s Juniper vMX counterpart). Each of this VMs require some memory to run, for the simplicity, lets say each VM requires 5GB of RAM.
So roughly, the above topology will claim 30-35GB of RAM in order to operate. Enriching the topology by adding more VMs of the same type will continue to push for more memory, thus running big topologies often becomes an exercise of hunting for RAM.
Luckily, there are technologies like Kernel Same Merging (KSM) and it’s enhanced version Ultra-KSM (UKSM) that are able to lift the memory requirement for use cases like above. In a nutshell, they allow to merge mem pages of the same content, effectively reusing the same memory pages between virtual machines.
from UKSM usenix paper
Memory deduplication can reduce memory footprint by eliminating redundant pages. This is particularly true when similar OSes/applications/data are used across different VMs. Essentially, memory deduplication detects those redundant pages, and merges them by enabling transparent page sharing.
Although UKSM is not a silver bullet for every application and use case, it tends to be a very good fit for hypervisors used to run virtualized networking topologies. For that reason the EVE-NG network emulation platform embeds UKSM in their product.
So I decided to bring UKSM to my Ubuntu 20.04 VM that I use to launch virtualized routers and containers to witness the benefits/issues of having it.
The results look promising. Running 6 VMs with a system memory footprint of one is a solid memory optimization, especially considering that performance penalty is something we can bare in a lab where we mostly play with control plane features.
Now if you want to bring UKSM to your hypervisor you will need to jump through some hoops, as UKSM is a kernel feature that is not available as a module. This means that you need to build a kernel with UKSM enabled, and that might be a barrier too high for some of you. It was for me, until I spent a night trying multiple things until it worked, so let me share with you the process and the outcomes so that you can rip the benefits without having all the trouble of trial-and-error routine.
- Download UKSM patches
- Download kernel source
- Apply UKSM patch
- Build kernel
- Install kernel
As mentioned above, UKSM is a kernel feature and the way it is distributed nowadays is via
patch files that are available in this Github repo. So our first step is cloning this repo to get the patches for recent (4.x and 5.x) kernels. Easy start.
As the UKSM patches need to be applied to a kernel source code, we need to get one. Here things can get a tad complicated.
There are many different kernels out there:
- vanilla Linux kernels blessed by Linus himself
- distribution kernels (Debian, Ubuntu, Fedora, etc)
- third party kernels with the best hacks
The UKSM patches were created against the vanilla Linux kernel, but my Ubuntu VM runs a kernel that was produced by Ubuntu team.
Vanilla linux kernel uses X.Y.Z versioning. If anything is appended after X.Y.Z (like
-48-generic) in my case, it indicates that the kernel comes from a distributor (Ubuntu in my case).
Things that didn’t work:
1 At first I tried to download the original Linux kernel, but the build process failed without giving me a good explanation.
2 Download latest 5.4 kernel from Ubuntu - UKSM patch didn’t apply, as the code has changed apparently
After multiple rinse-repeat iterations I found out that I can take the Ubuntu kernel
5.4.0-48.52 as UKSM patch applies to it no problem and the build succeeds.
How did I get one? Oh, that is also something worth documenting, as the path to knowing it is paved with broken links and articles dated early 2000s. First, go here and check what tags/branches are available for Focal release of Ubuntu. Once the tag/branch name is found, pull this one only, to save on data transfer:
Fast forward 180MB of kernel source code and you have it in
focal directory. Next is patching.
To embed the UKSM code into the kernel code we need to use the patch utility.
In the UKSM repo we cloned in step 1 we have patch files per kernel MAJOR.MINOR version. As we downloaded the Ubuntu kernel 5.4.0-something, let’s try and apply the patch from
uksm-5.4.patch patch file.
patch command must not return any failures. If it does, do not proceed!
This is the most important step, the patch must apply cleanly, meaning that if you see any
FAILURE strings in its output (or
echo $? doesn’t return
0) it means the patch is not compatible with the kernel.
The tricky part was to find the
Ubuntu kernel+patch file combination that didn’t result in an error. For me the merry pair was
Ubuntu-5.4.0-48.52 + uksm-5.4.patch.
Once the patch is cleanly applied we build the kernel.
Here I feel obliged to say that it was my first kernel build, so the explanations are surely not technically correct, but it works, so why not sharing my view on it.
To build the kernel we fist need to create the build configuration file. As we use the kernel that we actually run (5.4.0-48) we can reuse the existing kernel configuration:
This command will run the config generation script and it prompted me that there is a UKSM config option added (as a result of a UKSM patch) and if I want to use it instead of the default KSM option. I typed
1 in the prompt confirming that I need UKSM to be an acting KSM feature. That is the only input that was needed.
After the config is made, start the build process:
40 minutes later I had four debian packages created:
Out of these four files I needed to install all but
Once this is done, update your grub config to have the new kernel load by default:
After the reboot, ensure that your new kernel is running by examining
uname -r output. It should match the new version.
Launch some VMs, and check the memory consumption as well as the number of sharing pages with
If you don’t want to build a kernel yourself (and one night lost I can see why), I packaged the deb files into a bare container which you can pull and copy the files from to install the kernel on your Ubuntu machine:
All you need to do is to start with step 5 and you should be all good. Thanks for tuning in!
There is a KSM kernel feature that allows you to achieve some memory sharing via a similar mechanisms. It can be that KSM will deliver a similar performance on your setup, and being included in your kernel by default it might be worth checking out.
The following resources will help you start with KSM: