VM Lifecycle & the Golden Snapshot

Every Dev Box ("box") in doppiobox is a Firecracker microVM — a real Linux kernel with its own memory, disk, and network, started in milliseconds. This page explains the states a box moves through (create, start, stop, delete, reset), the files that back a box on disk, and the centrepiece: the golden snapshot that turns "boot a VM and install Frappe from scratch" (minutes) into "clone a frozen, ready-to-go VM" (~3–20s).

Audience: a newcomer to the agent codebase. We cover both how it works and why it is built this way.

All code references below are to the Go agent under backbone/agent/internal/.

1. Who drives the lifecycle

Frappe (the control plane) never touches a VM directly. It writes tasks with an action string; the agent polls, claims one task at a time, and dispatches on that string.

The dispatch table is the heart of the lifecycle — every transition starts here:

backbone/agent/internal/worker/worker.go:180

func (w *Worker) execute(task *db.Task) error {
	switch task.Action {
	case "create":       return w.handleCreate(...)
	case "start":        return w.handleStart(...)
	case "stop":         return w.handleStop(...)
	case "delete":       return w.handleDelete(...)
	case "reset":        return w.handleReset(...)
	case "build_golden": return w.handleBuildGolden(...)
	case "build_image":  return w.handleBuildImage(...)
	default:             return fmt.Errorf("unknown action: %s", task.Action)
	}
}

Each handleX updates the DB status, reports progress back to Frappe, and calls into the executor — the runtime backend. The executor interface (backbone/agent/internal/executor/executor.go:11) is implemented by the Firecracker backend (firecracker.Executor). The action strings map to executor methods like so:

build_image builds the base rootfs (a Docker image exported to ext4) that golden builds boot from — that's the layer below this page; the handleBuildImage worker path is documented on page 04.

2. The lifecycle state machine

A box's observable state is computed, not stored as truth, by InspectState (firecracker.go:385): if the VMM process is alive → running; else if a per-VM snapshot dir exists → stopped; else if an overlay file exists → exited; else → `` (gone). The DB tracks its own status string (creating, running, stopped, deleted) that the worker keeps in sync.

Two transitions are worth calling out because they are not user-initiated:

• reset (firecracker.go:350) wipes all per-VM disk state and rebuilds the box from the golden snapshot, but keeps its identity — same slug, same IP, same Caddy routes (IP allocation and AddRoute are idempotent). It leaves the box running so a user mid-meltdown gets a clean box back without a manual Start. Think "factory reset, machine still on".
• auto-restart (worker.go:84 reconcileVMs, every 30s): if the DB says a box is running but the VMM process has died, the worker calls Start to bring it back. This is why Start must be bulletproof and stateless.

3. What a box is made of (the four artifacts)

A running box is backed by four things on the host. Knowing which survive stop vs delete is the whole mental model.

The split matters:

• Stop (firecracker.go:306) only kills the VMM process, removes the TAP device, and drops the Caddy route. The overlay and volume stay on disk. So a stopped box keeps both its system changes and the user's files.
• Remove (firecracker.go:320, the executor half of delete) deletes overlay, volume, snapshot dir, serial log, socket, and frees the IP. Nothing survives.

A note on the per-VM snapshot (why stopped no longer snapshots)

Originally stop would snapshot the running VM and start would resume it (fast, preserves RAM state). That path was abandoned. The comment on Start explains why (firecracker.go:229):

"Snapshot-restore-of-a-snapshot-restored VM has a reliable virtio-net hang: the guest's network goes silent after ResumeVM."

Because every box is itself born from the golden snapshot (section 4), resuming it from a second snapshot reliably wedges its network. So today Start always cold-boots from the overlay + volume (firecracker.go:235). The trade-off, per the same comment: cold boot is ~30s slower than a ~3s resume, but the box always comes back network-reachable, with disk state intact — only in-RAM state (running shells, unsaved buffers, tmux) is lost, exactly like a reboot. Start even deletes any stale per-VM snapshot to reclaim disk (firecracker.go:252).

4. The golden snapshot — the centrepiece

What a snapshot is

A Firecracker snapshot is a frozen capture of a running VM: its guest memory (a mem file) plus its device/CPU state (a snapshot file). Restore those two files into a fresh Firecracker process and the guest resumes at the exact instruction it was paused on — kernel booted, services running, caches warm. No boot. No init.

The agent's snapshot primitives are tiny (firecracker/snapshot.go):

• CreateSnapshot (snapshot.go:10) — PATCH /vm {state:Paused}, then PUT /snapshot/create with snapshot_type: "Full", writing snapshot + mem into a dir.
• LoadSnapshotPaused (snapshot.go:46) — PUT /snapshot/load pointing at those two files, with resume_vm: false (load but stay paused).
• ResumeVM (snapshot.go:67) — PATCH /vm {state:Resumed}.
• HasSnapshot (snapshot.go:75) — checks both snapshot and mem exist.

Full vs diff snapshot: the code only ever uses snapshot_type: "Full" (snapshot.go:25) and explicitly sets enable_diff_snapshots: false on load (snapshot.go:56). Firecracker also supports diff snapshots (only changed memory pages), but doppiobox does not use them — every snapshot is the complete memory image.

Why "golden": the problem it solves

Cold-booting a box and provisioning it from nothing is slow: boot the kernel, start MariaDB, run bench new-site, run bench build (a heavy webpack/esbuild step that needs extra RAM), start code-server. That's minutes, and every single box would pay it.

The golden snapshot pays that cost exactly once. We boot one VM, fully provision it (Frappe installed, site created, bench built, optional extra apps installed, code-server up), snapshot it at the moment it's perfectly ready, and store that snapshot at /opt/devbox/snapshots/golden-{template}/. Every new box is then restored from that frozen ready state instead of being built — turning minutes of setup into ~3–20s.

Per the project CLAUDE.md: "New VMs clone from golden (~20s), pool box claims restore from per-VM snapshot (~3s)."

4a. Building the golden snapshot (one-time, per template)

BuildGoldenSnapshotForTemplate (golden.go:83). Triggered by the build_golden action. It is serialized by goldenMu (golden.go:23) because golden build and golden restore share the same fixed "golden slug" overlay path and must never run concurrently.

Key details and the why behind them:

• More RAM for the build. The golden VM uses GoldenMemory (4 GB) instead of the normal box memory, because bench build peaks high (golden.go:128). Boxes restored from it run at normal memory.
• Fixed, deterministic resources. The build uses a short hashed "golden slug" (goldenBuildSlug, golden.go:51) so the TAP name tap-g… fits the 15-char kernel interface limit, and a reserved IP 10.0.255.254 (goldenIP, golden.go:26) outside the normal allocation pool.
• MMDS is configured before the snapshot. The /mmds/config call (which authorises eth0 to reach the metadata service) is baked into the snapshot; the /mmds data is just a placeholder here and gets replaced per-box on restore. The comment at golden.go:155 is explicit: without configuring MMDS pre-snapshot, restored VMs can't reach 169.254.169.254.
• The overlay becomes the clone template. After snapshotting, the golden overlay is renamed to {snapshotDir}/rootfs.ext4 (golden.go:263). Firecracker snapshots hardcode the exact drive paths used at build time, so every restore must present an identical-content rootfs at the expected path — this saved rootfs.ext4 is the master copy that gets cloned for each new box.

4b. Creating a box by restoring golden (the fast path)

CreateFromGoldenTemplate (golden.go:302). This is what create does whenever a golden snapshot exists for the template (CreateWithTemplate, firecracker.go:105). If none exists for the default template, it falls back to createColdBoot; for a non-default template with no golden, it errors out (no silent slow path).

Why this is fast and why the awkward "golden-slug then rename" dance exists:

• No boot, no provisioning. Restoring maps the frozen memory image straight into the guest. Frappe, MariaDB, and code-server are already running in that memory — that's the whole ~minutes → ~seconds win.
• Snapshots hardcode drive paths and the TAP name (golden.go:290 comment). So the restore first stages the cloned overlay/volume at the golden-slug paths and creates tap-{goldenSlug}, loads the snapshot, resumes, and only then renames the files and ip link … names the TAP to the real slug. Firecracker holds the file descriptors open, so renaming underneath it is safe. This is also why golden restores are serialized — there's only one golden-slug path to stage into.
• Load paused, inject, then resume. LoadSnapshotPaused + PUT /mmds + ResumeVM (golden.go:365–391) means the new box's real metadata (its slug, site/code URLs, env vars) is in place before the guest's first instruction runs — otherwise the box would wake up believing it's the golden placeholder.

5. New identity for a clone: network & MMDS re-config

A restored clone wakes up wearing the golden VM's clothes: the golden IP (10.0.255.254), the golden MAC, and golden MMDS metadata. If two boxes shared a MAC, the bridge couldn't tell them apart; if they shared an IP, routing breaks. So part of "create from golden" is handing the clone a fresh identity:

1. MMDS metadata — buildMMDSPayload(slug, domain, envVars) is PUT /mmds while the VM is still paused (golden.go:375), replacing the baked-in golden-placeholder with this box's real slug/URLs/env. The guest reads this from 169.254.169.254 on resume.
2. IP + MAC — fixupNetwork(goldenIP, newIP) (golden.go:424) SSHes into the box (reachable at the golden IP right after resume) and rewrites eth0: a unique MAC derived from the new IP (GuestMAC), the allocated IP added, then the old golden IP deleted in the background — because deleting the IP you're SSH'd in over would kill the connection mid-command (see the nohup … sleep 0.5; ip addr del trick at golden.go:446).
3. Caddy route + env injection — AddRoute points {slug}.site/.code.{domain} at the new IP; injectEnvVars writes /etc/profile.d/devbox.sh and restarts code-server so URLs resolve.

The MMDS configuration that makes step 1 possible (authorising eth0) was baked into the snapshot back in section 4a. The full MMDS metadata contract and the network/MAC derivation scheme are detailed on page 05.

6. Warm pool

The project CLAUDE.md distinguishes two restore speeds: "New VMs clone from golden (~20s), pool box claims restore from per-VM snapshot (~3s)." This implies a warm pool — boxes pre-created and parked, ready to be claimed and handed to a user near-instantly, with the cold-clone work already done ahead of time.

Note: there is no warm-pool / claim implementation in the agent code. A full sweep of internal/ finds only ClaimPendingTask (the task-queue claim, unrelated) and the CreateFromGolden comment "pool creation is sequential" (golden.go:296). The pre-create/claim orchestration therefore lives Frappe-side, not in this agent. Treat the ~3s "pool box claim" figure as documented intent — the agent itself only does the ~3–20s golden clone described above.

7. Cheat-sheet