Introduction

After two weeks of almost up to 37°C here in Germany my brain is now able to work normally again. I'm ready for new hot stuff.

Weaveworks turns 5 this month (HAPPY BIRTHDAY!) and they're celebrating with a release of something called ignite. In this post I will try to answer the questions about what it is, how it works and who needs it. I'll also add a walkthrough/ demo about how to run ignite!

What

Ignite is a tool for managing microVMs in the style of managing Docker containers. For people who are familiar with the easy usage of the Docker CLI they will feel like "home". If you're familiar with the commands like run, kill, logs, ps, rm you'll be able to manage your own microVMs. From the state of running all your services in containers according to the benefits of flexibility, scalability, isolation, fast integration and delivery, ignite enforces you to run your services in real isolated virtual machines with the same benefits from containers, with a minimal footprint and better security and stability. Ignite bootstraps VMs directly on your linux host based on the KVM (Kernel-based Virtual Machine) technology which was introduced in linux kernel 2.6.0 in 2006. Ignite uses the from AWS (Amazon Web Services) developed Firecracker project to create and manage virtual machines on KVM. More about Firecracker later on.

It wouldn't be a weaveworks project if they weren't implementing their GitOps workflow. Here is a nice article which explains GitOps detailed: What is GitOps?. tl:dr Git is the source of truth. Operate your infrastructure, systems, services with git. If something changes on your stable branch, it automatically gets rollout. This can be implemented in a pull or push principle. Every change to the repository is tracked by git and makes reproducibility and rollback so easy! I'll also wrap up this feature later on!

How

Before I explain how ignite and Firecracker works, I want to bring on mind the difference between linux containers and VMs based on KVM. Maybe the wording "run your container workload on real VMs" is a little bit confusing (Buzzword bingo).

Container vs KVM

If you didn't know already Docker is not the inventor for container virtualisation. Docker is an container manager which makes it very easy to bootstrap/manage containers. When we talk about containers we're actually talking about linux namespaces. tl:dr linux namespaces are isolated processes on your system which run in PID 1. Child processes of this isolated namespace can only see their parent process. PID 1 have the parent PID of 0. There is more to talk about namespace networking and namspace mounts but this would blow up this post.

Containers can directly talk to the hosts system kernel, so that they don't need an own kernel. They only require a linux filesystem(filesystem bundle) which includes a init-system like systemd. This is the reason why containers are so lightweight.

With KVM it is possible to create complete independent and isolated virtual machines on the same host. Each virtual machine requires a kernel and linux filesystem. All VMs have also access to the underlying host resources, but they're limited on creation of the virtual machine.

Virtual machines are more secure than containers. If a container gets access to the underlying host it will be able to operate maliciously. Containers can gain host access through a bug / CVE or more simple with privileged permissions. With Docker you can easily permit fully host access with the --privilegedflag. In a virtual machine a malicious operation can not affect other VMs, because they do not know about each other.

Wouldn't it be nice to get both benefits from containers and VMs in a surefire way? Let's see how Firecracker may be helping us to get this.

Firecracker

Firecracker is a VMM (Virtual Machine Manager) for managing KVM-Hypervisors. It's developed by the AWS cloud-provider. The intension of this project is to create secure, multi-tenant, minimal-overhead execution of container and function workloads - GitHub. AWS uses Firecracker for their serverless service Lambda.

Firecracker manages the complete life cycle of virtual machines on the host and adds further security boundaries. Please read the architecture section of firecracker for a deeper overview. Firecracker is completely manageable through it's delivered REST-API. For the state of a modern VMM it also delivers Prometheus metrics about the running microVM.

So what does firecracker require to run an microVM?

  • Linux Host with KVM
  • A linux kernel for the VM - Interface between hardware and system processes, starts the required system services from the linux filesystem under /sbin/init
  • A linux file system / file bundle with an init-system - starts all required system services. One of the most famous init-system is systemd

Ignite

Ignite enforces us to build read-only snapshot images for Firecracker microVMs with custom kernels and custom OCI (Open Container Initiative) conform images like Docker images. OCI conforms images describes the required design of the linux-filesystem / file-bundle. Luckily Docker images adopt this conformation. For more information about OCI images please have a look here.

This enables developers to run our existing container-images in an complete isolated virtual-machines without complaining about the file-system design. I think this is the biggest benefit of ignite. You as a developer are now able to create your own custom VM images.

I mean how cool is that!

From the ignite repository you can get adetailed overview of how it creates microVMs. Here is a short explanation:

  • Build a read-only VM snapshots with the provided linux kernel and OCI image
  • Start a container with PID 1 with Firecracker inside with privileged permission
  • Firecracker use the custom built image and starts the microVM on your KVM Host
  • Ignite forwards the container network to the microVM so it is accessible over the containers IP

overview_1-2

Walkthrough

After wrapping up the fundamentally basics of how ignite works, let's start our walkthrough how to use it. You will see it is totally simple!

Please following the official install instruction and check if you have all requirements: weaveworks/ignite/install.

By default ignite uses its own kernel from weaveworks which comes in an container and gets uncompressed on your system: weaveworks/ignite-kernel:4.19.47. Now we need an OCI Image which will run on top of the kernel. Weaveworks also provides images for this.

First step is to import this image to ignite before we can use it. If the image is not present on the host ignite will pull it.
At this point you can import your own custom container image for your needs.

$ sudo ignite image import weaveworks/ignite-ubuntu

Like from the Docker CLI you can inspect the current images on your host with the images command.

$ sudo ignite images
IMAGE ID		NAME				CREATED		SIZE
2601d09c43c5adf9	weaveworks/ignite-ubuntu:latest	1m ago	220.5 MB

Now we are ready to create a VM and start it.

$ sudo ignite create weaveworks/ignite-ubuntu \
	--name demo-vm \
	--cpus 2 \
	--memory 4GB \
	--size 3GB \
	--ssh

INFO[0000] Imported OCI image "weaveworks/ignite-kernel:4.19.47" (49.0 MB) to kernel image with UID "6acc6369b0c00d42" 

And we can also check if the microVM was created with vm.

$ sudo ignite vm

VM ID			IMAGE				KERNEL					CREATED	SIZE	CPUS	MEMORY	STATE	IPS	PORTS	NAME
8e6d14da33670697	weaveworks/ignite-ubuntu:latest	weaveworks/ignite-kernel:4.19.47	53s ago	3.0 GB	2	4.0 GB	Created			demo-vm

Before we start the VM let's inspect the Firecracker directory on your host.

$ ls  /var/lib/firecracker
image  kernel  vm

The image folder includes the filesystem of the imported image and metadata information about it.

$ ls  /var/lib/firecracker/image/16af637676a07c19/
image.ext4  metadata.json

$ cat  /var/lib/firecracker/image/16af637676a07c19/metadata.json 
{
  "kind": "Image",
  "apiVersion": "ignite.weave.works/v1alpha1",
  "metadata": {
    "name": "weaveworks/ignite-ubuntu:latest",
    "uid": "16af637676a07c19",
    "created": "2019-07-15T21:00:41Z"
  },
  "spec": {
    "ociClaim": {
      "type": "Docker",
      "ref": "weaveworks/ignite-ubuntu:latest"
    }
  },
  "status": {
    "ociSource": {
      "id": "sha256:f8032a6738fdb372eb7e64734c9481f873b57e0ea72b610e0189f0a329283173",
      "size": "231256502B",
      "repoDigests": [
        "weaveworks/ignite-ubuntu@sha256:ad984fa5f6f2db55a0a48860d263a02a0f77aee3bbefa0d71648b4bc287ac13c"
      ]
    }
  }
}

In the kernel folder you can find all imported kernels and in the vm folder your created VMs.

$ ls  /var/lib/firecracker/vm/8e6d14da33670697/
id_8e6d14da33670697  id_8e6d14da33670697.pub  metadata.json  overlay.dm

$ cat  /var/lib/firecracker/vm/8e6d14da33670697/metadata.json
{
  "kind": "VM",
  "apiVersion": "ignite.weave.works/v1alpha1",
  "metadata": {
    "name": "demo-vm",
    "uid": "8e6d14da33670697",
    "created": "2019-07-15T21:02:44Z"
  },
  "spec": {
    "image": {
      "ociClaim": {
        "type": "Docker",
        "ref": "weaveworks/ignite-ubuntu:latest"
      }
    },
    "kernel": {
      "ociClaim": {
        "type": "Docker",
        "ref": "weaveworks/ignite-kernel:4.19.47"
      },
      "cmdLine": "console=ttyS0 reboot=k panic=1 pci=off ip=dhcp"
    },
    "cpus": 2,
    "memory": "4GB",
    "diskSize": "3GB",
    "network": {
      "mode": "docker-bridge"
    },
    "ssh": true
  },
  "status": {
    "state": "Created",
    "image": {
      "id": "sha256:f8032a6738fdb372eb7e64734c9481f873b57e0ea72b610e0189f0a329283173",
      "size": "231256502B",
      "repoDigests": [
        "weaveworks/ignite-ubuntu@sha256:ad984fa5f6f2db55a0a48860d263a02a0f77aee3bbefa0d71648b4bc287ac13c"
      ]
    },
    "kernel": {
      "id": "sha256:5e5165af3e5dda9371abce633189eed1f80f9c493635e37264c6783fd5e0f6c9",
      "size": "51377512B",
      "repoDigests": [
        "weaveworks/ignite-kernel@sha256:18ee1005bb3881faf09059038eca8b298d9d7598cde675c4a900e275fe0a2454"
      ]
    }
  }
}

Now lets start our created microVM and see what happens.

$ sudo ignite start demo-vm
INFO[0001] Started Firecracker VM "8e6d14da33670697" in a container with ID "3a207a02945fbc220e84250f15285442219f034d77e9d497c2b1a500f76c3cef" 

Ignite started now a container with Firecracker inside to manage the microVM on the host. For each new microVM ignite starts a new container.

$ docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED              STATUS              PORTS               NAMES
3a207a02945f        weaveworks/ignite:v0.4.0   "/usr/local/bin/igni…"   About a minute ago   Up About a minute                       ignite-8e6d14da33670697

The microVM is up and running. We have now two options to connect to it. Because we added the --ssh flag at creation we are able to connect via ssh. Without this flag you can use sudo ignite attach.

$ sudo ignite ssh demo-vm

Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.19.47 x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage
This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, you can run the 'unminimize' command.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@8e6d14da33670697:~# 

Now let's get some information about the microVM:

root@8e6d14da33670697:~# cat /etc/os-release 
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

root@8e6d14da33670697:~ ip address 
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 06:c5:f8:7b:a6:79 brd ff:ff:ff:ff:ff:ff
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether ae:21:4b:14:ec:54 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::ac21:4bff:fe14:ec54/64 scope link 
       valid_lft forever preferred_lft forever

We can see that the microVM got an address from the hosts Docker network.

To exit the microVM just type exit or to shut it down reboot. With the ps command we can see the current running mircoVMs.

root@8e6d14da33670697:~# exit
logout
Connection to 172.17.0.2 closed.

$ sudo ignite ps
VM ID			IMAGE				KERNEL					CREATED	SIZE	CPUS	MEMORY	STATE	IPS		PORTS	NAME
8e6d14da33670697	weaveworks/ignite-ubuntu:latest	weaveworks/ignite-kernel:4.19.47	34m ago	3.0 GB	2	4.0 GB	Running	172.17.0.2		demo-vm

We also can inspect the microVM logs.

$ sudo ignite logs demo-vm
...
[  OK  ] Started Network Name Resolution.
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Serial Getty on ttyS0.
[  OK  ] Started Getty on tty1.
[  OK  ] Reached target Login Prompts.
[  OK  ] Reached target Host and Network Name Lookups.
[  OK  ] Started OpenBSD Secure Shell server.
[  OK  ] Started Dispatcher daemon for systemd-networkd.
[  OK  ] Reached target Multi-User System.
[  OK  ] Reached target Graphical Interface.
         Starting Update UTMP about System Runlevel Changes...
[  OK  ] Started Update UTMP about System Runlevel Changes.

Ubuntu 18.04.2 LTS 8e6d14da33670697 ttyS0

You can stop the microVM with sudo ignite stop demo vm and remove it from your host with sudo ignite rm demo-vm.

The GitOps way

As I mentioned above ignite also includes GitOps. This means you can manage microVMs with git. In yaml formated files you describe these with all your requirements of resources, hostname, ssh, image and current state. All you need for this is a public accessible git repository. Everything else ignite will do.

Here is the example yaml from the ignite repository which we will use:

apiVersion: ignite.weave.works/v1alpha1
kind: VM
metadata:
  name: ubuntu-vm
  uid: 05286f8fb73dc324
spec:
  cpus: 2
  diskSize: 4GB
  memory: 1GB
  image:
    ociClaim:
      ref: weaveworks/ignite-ubuntu:latest
  kernel:
    ociClaim:
      ref: weaveworks/ignite-kernel:4.19.47
  ssh: true
status:
  state: Created

Please have a look on the current state. We will change it from Created to Running in a separate commit and hopefully can see that ingite will start the microVM automatically.

Before pushing this config to the public git repository, start sudo ignite gitops <URL> so it starts watching.

sudo ignite gitops https://github.com/fwiedmann/ignite-demo
INFO[0000] Starting GitOps loop for repo at "https://github.com/fwiedmann/ignite-demo" 
INFO[0000] Whenever changes are pushed to the master branch, Ignite will apply the desired state locally 
INFO[0000] Initializing the Git repo...                 
INFO[0010] Git initialized: A bare clone of repo "https://github.com/fwiedmann/ignite-demo" has been made

Ignite is now watching on your repository. Now push the microVM description. Ignite recognized this and creates the mircoVM.

INFO[0010] New commit observed on branch "master": b84d142f0035f60e54f2cad993f9f351f40c59a7 
INFO[0010] File "test.yaml" was created. It describes a VM with UID "05286f8fb73dc324" and name "ubuntu-vm" 
INFO[0015] Waiting for updates in the Git repo...       
INFO[0015] Creating VM "05286f8fb73dc324" with name "ubuntu-vm"... 


$ sudo ignite ps -a
VM ID			IMAGE				KERNEL					CREATED		SIZE	CPUS	MEMORY		STATE	IPS	PORTS	NAME
05286f8fb73dc324	weaveworks/ignite-ubuntu:latest	weaveworks/ignite-kernel:4.19.47	3m57s ago	4.0 GB	2	1024.0 MB	Created			ubuntu-vm

The microVM is not running yet, because we set the status to the state of Created. Now let's change the status to Running.

INFO[0010] New commit observed on branch "master": d9463b7e219c466d7ebf5ee7ddcc489df9f1bba4 
INFO[0010] File "test.yaml" was created. It describes a VM with UID "05286f8fb73dc324" and name "ubuntu-vm" 
INFO[0015] Waiting for updates in the Git repo...       
INFO[0015] Starting VM "05286f8fb73dc324" with name "ubuntu-vm"... 

$ sudo ignite ps
VM ID			IMAGE				KERNEL					CREATED	SIZE	CPUS	MEMORY	STATE	IPS		PORTS	NAME
8e6d14da33670697	weaveworks/ignite-ubuntu:latest	weaveworks/ignite-kernel:4.19.47	34m ago	3.0 GB	2	4.0 GB	Running	172.17.0.2		ubuntu-vm

Works as desired :)

Who needs this

In my opinion ignite can be helpful for everyone who needs the benefits of microVMs. It works perfectly for local development of your workload but much more with the GitOps feature. It fits perfectly in my everything as code philosophy. I hope also in yours!

At this point feel free to explore ignite by yourself. I hope this little introduction was helpful for you.

Big thanks to weaveworks for your great work! Looking forward to release 1.0.0 of ignite!

If you have any questions please feel free to contact me on twitter.