Firecracker Explained (microVMs, and why not Docker)

Audience: newcomer. This page teaches Firecracker from first principles, then explains why benchspace/doppiobox runs every dev box inside a Firecracker microVM instead of a Docker container. By the end you should understand what a microVM is, why it gives stronger isolation than a container, and how the agent actually boots one.

1. Start here: what problem are we solving?

A doppiobox dev box is a full Frappe development environment a stranger on the internet pays for and then gets root inside. They open a terminal in VS Code, run arbitrary code, sudo, install packages, compile things. Many such users share one physical server.

That is the textbook definition of a hostile multi-tenant workload: untrusted code, root-level, packed many-to-a-host. The single most important question for the runtime is therefore:

If a user breaks out of their box, can they reach the host or another user's box?

Everything below is about answering "no" to that question while still booting fast and packing many boxes onto one machine.

2. What is Firecracker?

Firecracker is a Virtual Machine Monitor (VMM) built by AWS, written in Rust, that boots stripped-down virtual machines called "microVMs". It is the engine underneath AWS Lambda and AWS Fargate, so it is battle-tested at enormous scale.

Three terms a newcomer needs:

• VMM (Virtual Machine Monitor / hypervisor) — the host-side program that creates a virtual machine, gives it virtual CPUs and memory, and emulates a minimal set of virtual devices (a disk, a network card). Firecracker is that program. One Firecracker process on the host == one running guest VM.
• KVM (Kernel-based Virtual Machine) — a feature built into the Linux host kernel that exposes the CPU's hardware virtualization extensions (Intel VT-x / AMD-V) via the device /dev/kvm. KVM is what lets a guest run its own kernel directly on the physical CPU at near-native speed, with the CPU hardware itself enforcing the boundary between guest and host. Firecracker is a thin userspace driver on top of KVM — it does not implement virtualization itself; it asks the hardware (via KVM) to do it.
• microVM — a deliberately minimal VM. A normal VM (think VirtualBox/QEMU) emulates a whole PC: BIOS, PCI bus, USB, video, sound, dozens of devices. Firecracker throws almost all of that away and offers only what a server workload needs: a serial console, a virtio block device (disk), a virtio network device, and a couple more. Less emulated hardware means less code, faster boot, less memory, and a smaller attack surface.

The payoff of "minimal": a Firecracker microVM boots in ~125 ms and the VMM process itself adds only a few MB of overhead — yet each microVM is a real virtual machine running its own Linux kernel, isolated from the host by the CPU's virtualization hardware.

Because /dev/kvm is hardware virtualization, Firecracker only runs on bare metal (or a host that exposes nested virtualization). This is exactly why the migration bought a Hetzner AX42 bare-metal server — most cloud VMs do not expose /dev/kvm. The agent refuses to start the Firecracker backend if /dev/kvm is missing (firecracker.go:48).

Note: ~125 ms is the canonical Firecracker VMM-to-kernel figure. On the Frappe rootfs the validation doc measured ~5s "cold boot to shell" — that includes the guest kernel + systemd + services starting, not just the VMM handing control to the kernel.

3. The core comparison: microVM vs Docker container

This is the heart of the page. Both Docker and Firecracker "isolate a workload", but they draw the security boundary in fundamentally different places.

How a Docker container isolates

A container is not a machine. It is a normal Linux process on the host that the kernel has been told to show a restricted view of the world. Two kernel features do this:

• namespaces — give the process its own private view of PIDs, mounts, network interfaces, hostnames, users. (Why it looks like its own machine.)
• cgroups — cap how much CPU / memory / IO the process may consume. (Why it can't hog the host.)

The critical fact: all containers share the one host kernel. When code inside a container makes a syscall, it is talking directly to the host's kernel. There is no second kernel in between. The isolation is "the same kernel pretends to be many machines."

So the attack surface is the entire Linux system-call interface — hundreds of syscalls. A single kernel-level vulnerability (a bad syscall, a namespace escape, a cgroup bug) lets a container break out and become root on the host, which means root over every other tenant. This is the well-known class of "container escape" CVEs.

How a Firecracker microVM isolates

The guest runs its own, separate Linux kernel. User code inside the box makes syscalls to the guest kernel — never to the host kernel. The only thing the guest can talk to on the outside is the tiny set of virtual devices Firecracker emulates, and that conversation happens through KVM with the CPU hardware enforcing the wall.

For a guest to reach the host it would have to break out of the CPU's hardware-virtualization boundary and then exploit Firecracker's deliberately tiny device-emulation surface (which is further sandboxed with seccomp). That is a dramatically harder and smaller target than "any syscall in the host kernel".

Side-by-side

Read the left side as: three boxes, one wall (the host kernel). Breach it once, own everything. Read the right side as: three boxes, each with its own kernel, sitting behind a hardware wall (KVM). A breach inside a box stops at that box's kernel.

Comparison table

Be fair about the trade: Docker is simpler to operate, runs anywhere, and has nicer day-to-day tooling. Firecracker costs you bare-metal hosting and a chunk of operational complexity (TAP devices, bridges, NAT, kernel/rootfs management). doppiobox pays that price on purpose — see next.

4. Why benchspace chose Firecracker for dev boxes

The migration plan (plans/firecracker-migration.md) states the rationale directly. Paraphrasing its "Context" and "Benefits" sections:

1. Stronger isolation for untrusted, root-level user code. This is the headline. Containers share the host kernel — "a weaker security boundary for multi-tenant workloads." Firecracker gives each box its own kernel via KVM, so a box escape "doesn't compromise the host." When users get root in their box, this is non-negotiable.
2. Fast boot + snapshot-powered pool. A microVM cold-boots in ~125 ms, and — bigger win — a box can be restored from a memory snapshot in sub-second time. That turns "pool box claim" from minutes (full container create + first-run Frappe init) into seconds. (Snapshots are page 03.)
3. True (hard) resource limits. KVM caps are physical, not advisory cgroup limits. A box configured for 2 GB cannot touch the host's RAM beyond that.
4. Memory efficiency via kill-and-restore. Stopping a box kills the VMM process, returning every byte of its RAM to the host immediately. A stopped box costs 0 MB. (This avoids the memory-never-returned trap the plan cites from the Hocus.dev post-mortem.)
5. Higher density / better economics. The plan's table projects ~30 boxes on one ~€48/mo bare-metal box (cost per slot ~$1.77) vs ~10–12 on the old ~$59/mo cloud Docker host (~$5.90/slot).

The plan keeps the rest of the system identical: same pull-based agent, same Frappe backend API, same user-facing URLs. Firecracker is swapped in behind an executor.Executor interface so Docker and Firecracker backends coexist and can be chosen per-server. That interface is why this page's code lives in internal/firecracker/ next to internal/docker/, both implementing the same Go interface.

5. The building blocks of one microVM

Before walking the boot code, here are the physical pieces. Host paths come from CLAUDE.md and the config defaults in backbone/agent/internal/config/config.go.

Why "copy-on-write rootfs" matters

The base image is read-only and shared; each box only stores the blocks it changes (sparse copy). That keeps per-box disk small while still giving every box a fully writable, independent /. The migration plan notes a future optimization (squashfs read-only base + per-VM overlayfs) to shave ~4 GB/VM, but the current, validated approach is the plain sparse cp you see in the code.

A note on the jailer

Firecracker ships a companion binary called the jailer. Conceptually it is a hardening wrapper you launch instead of launching Firecracker directly: it chroots the Firecracker process into a per-VM directory, drops privileges (runs it as an unprivileged uid/gid), and places it in its own cgroup and namespaces. That's "defense in depth" — even if someone broke out of the KVM boundary into the Firecracker process, the jailer ensures that process is itself a caged, unprivileged thing.

Honest status in this codebase: the jailer is planned and configured — there's a JailerPath default of /opt/devbox/bin/jailer (config.go:31, :93) and the migration plan describes booting "via jailer". But the code that actually runs today launches Firecracker directly, not through the jailer: startFirecracker runs exec.Command(e.cfg.FirecrackerPath, "--api-sock", sock) with no jailer wrapper (firecracker.go:617). The JailerPath config field is never read by any call site — it has no constructor or exec.Command use anywhere in internal/. So treat the jailer as the intended defense-in-depth layer, not as something currently in the boot path.

6. How a VM is actually launched (walking firecracker.go)

The interesting path is createColdBoot (firecracker.go:121) — provisioning a box from scratch. (The fast path, CreateFromGoldenTemplate, restores a snapshot instead; that's page 03.) Here's what it does, in order:

1. Allocate an IP from the 10.0.0.0/16 pool, tracked in SQLite (db.AllocateIP, line 125). Each box gets one address like 10.0.0.7.

2. Copy the rootfs → overlays/{slug}.ext4, the box's writable / (sparse copy, line 132).

3. Create the workspace → a fresh 30 GB sparse ext4 (createSparseExt4, line 140).

4. Create the TAP device and attach it to the br-devbox bridge (CreateTAP, line 147 → network.go).

5. Start the Firecracker process (startFirecracker, line 157). This spawns firecracker --api-sock <sock>, points its stdout/stderr at the serial-log file, puts it in its own process group, and waits for the API socket to appear before continuing (firecracker.go:608). At this moment the VMM is alive but the guest has not booted — Firecracker is just sitting on its socket waiting for configuration.

6. Derive the guest MAC and kernel boot args. The MAC is deterministic from the IP (GuestMAC, network.go:37). The boot args (BootArgs, network.go:48) are the kernel command line:

   console=ttyS0 reboot=k panic=1 pci=off ip=<ip>::10.0.0.1:255.255.0.0::eth0:off
   

   Read that as: log to the serial console; on panic, reboot (don't hang); disable PCI probing (the microVM has no PCI bus — this speeds boot and shrinks surface); and configure the NIC's static IP/gateway/mask at kernel level via the ip= directive, so no DHCP client is needed.

7. Configure the VM over the API socket, then start it. The code builds an ordered list of PUTs (firecracker.go:190-202) and fires them one by one. This is the machine definition — here is the real config:

   API endpoint	What it sets	Real value from the code
   /boot-source	Kernel + command line	kernel_image_path = vmlinux, boot_args = the string above
   /drives/rootfs	Root disk → /dev/vda	the per-VM overlay; is_root_device: true, writable
   /drives/workspace	Second disk → /dev/vdb	the workspace volume; not root, writable
   /network-interfaces/eth0	The guest NIC	guest_mac = derived MAC, host_dev_name = tap-{slug}
   /machine-config	CPU + RAM (the hard limits)	vcpu_count = e.cfg.CPUs (default 2), mem_size_mib = parsed from Memory (e.g. 2g → 2048)
   /mmds/config + /mmds	Metadata service the guest can read at 169.254.169.254 (env vars, slug, domain)	configured on eth0, version V2
   /actions	Boot it	{"action_type":"InstanceStart"}

   Only after that final InstanceStart does the guest kernel actually begin executing. If any step fails, the code tears everything back down (kill VMM, remove TAP, free IP, delete the disk files) — note the repeated cleanup blocks.

8. Wire up routing + env. Add the Caddy route so {slug}.site.{domain} reaches the box (line 216), and inject env vars (line 221). Returns an id like fc-<pid>.

Anatomy of one running microVM

7. Where Docker still appears (layering)

A newcomer might assume "Firecracker replaced Docker, so Docker is gone." Not quite — Docker survives in two distinct, outside-the-VM roles. Crucially, nothing runs Docker inside a Firecracker box. Firecracker is the only isolation layer for user code.

Role 1 — Build-time rootfs factory. The base rootfs is built with Docker, then thrown away. backbone/scratch/images/build-rootfs.sh does: docker build the Frappe image → docker create a container → docker export its filesystem to a tar → unpack the tar into an ext4 image. Docker is used purely as a convenient way to assemble a filesystem; the result is a plain ext4 file Firecracker boots. By the time a box runs, Docker is nowhere in the picture. (This is why docker export gotchas — like it stripping /etc/hosts — show up in the validation doc.)

Role 2 — The alternative backend (different servers). backbone/agent/internal/docker/docker.go is a peer of the Firecracker executor, not a layer beneath it. It implements the same executor.Executor interface and runs boxes as plain Docker containers on the old cloud hosts. main.go picks one backend per server from config (backend: docker or backend: firecracker). A given box is either a Docker container or a Firecracker microVM — never both. The migration is a gradual per-server cutover; Docker boxes drain as users delete them.

So the accurate mental model is:

• On a Firecracker server: user code → Firecracker microVM (own kernel, KVM). No Docker involved at runtime. Docker only ever ran earlier, on the build machine, to bake the rootfs.
• On a legacy Docker server: user code → Docker container (shared host kernel). No Firecracker.

There is no "Docker inside Firecracker" nesting in this system. (Such a thing exists in the wild — e.g. firecracker-containerd runs OCI containers inside microVMs — but doppiobox does not do it. The plan lists it only as a reference, not an architecture.) Confirmed in the code: docker is only invoked at build time (worker.go exportDockerToExt4: docker create → docker export → unpack to ext4) and by the standalone docker.Executor backend — never inside a guest's init path.

8. Where to go next

• Page 03 — Snapshots & the golden pool: how snapshot.go saves memory+device state and restores a box in sub-second time, and how the "golden snapshot" pre-warms the pool.
• Page 05 — Networking: the full story of network.go — TAP devices, the br-devbox bridge, deterministic MACs, kernel ip= config, and NAT to the internet.

Quick reference: facts to remember

• Firecracker = AWS's Rust VMM that boots minimal microVMs on KVM; ~125 ms boot, few-MB overhead, its own kernel per box.
• Container isolation = one shared host kernel (namespaces+cgroups). microVM isolation = per-box kernel behind hardware (KVM). The latter is why an escape stays contained — the whole reason doppiobox uses it for untrusted, root-level user boxes.
• One VM = one firecracker process + one API socket + vmlinux (shared) + per-VM overlay rootfs (/dev/vda) + workspace volume (/dev/vdb) + a tap-{slug} NIC.
• The machine is defined by a sequence of API PUTs ending in InstanceStart — see firecracker.go:190.
• The jailer is the intended defense-in-depth wrapper but is not yet in the live boot path (Firecracker is launched directly).
• Docker only builds the rootfs (build-time) and runs the legacy backend on other servers. It never runs inside a microVM.