The Go devbox-agent Internals

The local hands of the control plane. Pages 02/03/05 own the Firecracker / networking / snapshot machinery; this page owns the Go binary that drives them: how it boots, how it gets work, how it dispatches that work, and every knob and endpoint it exposes.

Audience: a Go-literate newcomer. We cover how and why, and verify every claim against the source under backbone/agent/.

1. What the agent is

devbox-agent is a single statically-linked Go binary that runs as a systemd service on each Hetzner server (one per server: fc-01 prod, the test box, etc.). It has exactly one job:

Be the local hands of the Frappe control plane. Frappe decides what should happen ("create a box named mybox", "delete oldbox"); the agent is the thing actually sitting on the server that makes it happen — booting Firecracker VMs, wiring Caddy routes, allocating IPs — and reports the result back.

Why pull, not push? Frappe never SSHes into servers to run commands. Instead each server runs this agent, which polls Frappe for work and reports back. That is simpler (no inbound SSH/credentials from the control plane), more secure (the server initiates all connections outbound), and survives network blips — when connectivity returns, the agent just resumes polling (backbone/agent/architecture.md:7).

The binary is backend-agnostic. It defines an Executor interface and picks an implementation at startup: firecracker (production) or docker (the older/local-dev path). Everything above the Executor boundary — polling, the task queue, the HTTP API — is identical regardless of backend (backbone/agent/cmd/devbox-agent/main.go:59).

Stale-doc warning: backbone/agent/architecture.md describes the original Docker + Traefik design and a "retry ≤ 3" task policy (verified: its flow diagram still reads "Execute (docker create/stop/rm)" and "Mark done or failed (retry ≤ 3)"). The live code has moved to Firecracker + Caddy and does not auto-retry failed tasks (see §6). Where they disagree, the code wins.

2. Package / component map

3. Process startup (main.go)

Step by step (main.go:23–137):

1. Config — config.Load(*configPath) reads the YAML, applies defaults, and overlays env vars. Fatal on error (main.go:28).
2. Logging — a slog JSON handler writes to stdout (journald captures it), level chosen from cfg.Agent.LogLevel (main.go:34).
3. Database — db.Open opens SQLite at /opt/devbox/agent/devbox-agent.db, enables WAL, runs migrations (main.go:51). Closed on shutdown via defer.
4. Executor — switch on cfg.Agent.Backend: firecracker → microVM executor, anything else → Docker executor (main.go:59). The Firecracker constructor gets the DB handle too, because IP allocation lives in SQLite (firecracker.go:40).
5. Frappe source — frappe.New(cfg.Frappe) returns an HTTPClient if a URL is set, else a StubSource no-op (main.go:78).
6. Firecracker-only startup tasks — if the backend is Firecracker, the agent fetches the Cloudflare API token from Frappe and configures wildcard TLS, then calls ReconcileRoutes() to re-register Caddy routes for VMs that were already running before this restart (main.go:81). This is what makes an agent restart non-disruptive to live boxes.
7. Worker + API are constructed; the SSH key path is injected into the worker (used when building rootfs images) (main.go:96).
8. Goroutines — go work.Run(ctx) and go httpServer.ListenAndServe() run concurrently (main.go:105, main.go:113).
9. Shutdown — block until SIGINT/SIGTERM, then cancel() the context (the worker's loop sees ctx.Done() and returns) and httpServer.Shutdown with a 5s grace window (main.go:122). KillMode=process in the unit file means systemd kills only the agent, not the child Firecracker VMs (devbox-agent.service:17).

4. The Executor contract

The worker never imports firecracker or docker types directly — it talks to this interface, which is why the same orchestration code drives both backends (executor.go:11):

type Executor interface {
    Create(slug string, envVars map[string]string) (string, error)
    Start(slug string) error
    Stop(slug string) error
    Remove(slug string) error
    RemoveVolume(slug string) error
    InspectState(slug string) (string, error)
    InspectID(slug string) (string, error)
    Logs(ctx, slug, follow) (io.ReadCloser, error)
    Stats(slug string) (*ContainerStats, error)
    VolumePath(slug string) string
    WorkspaceTarStream(ctx, slug) (io.ReadCloser, error)
    WaitForReady(ctx, slug, port, timeout, interval) error
    PlainTextLogs() bool
    Close() error
}

Some capabilities are optional and discovered at runtime via Go type assertions, so a backend that lacks them simply isn't offered the action:

• TemplateAwareExecutor — CreateWithTemplate, BuildGoldenSnapshotForTemplate (worker.go:208).
• ResettableExecutor — Reset(slug, templateSlug, envVars) (worker.go:215).
• GoldenSnapshotBuilder — the four golden-snapshot methods the API checks for (api.go:205).

This is the seam between this page and pages 02/03/05: the worker calls w.exec.Create(...); what that does — jailer, kernel, overlay, TAP device, Caddy route — is the Firecracker package's story.

5. The Frappe client (internal/frappe)

TaskSource is the whole contract between agent and control plane (client.go:29):

type TaskSource interface {
    Poll(server string) ([]FrappeTask, error)
    ReportStatus(jobName, status, errorMsg string) error
    ReportProgress(jobName, step string) error
    FetchServerConfig() (ServerConfig, error)
}

frappe.New returns HTTPClient when cfg.Frappe.URL is set, otherwise StubSource (all methods return nil/empty — standalone mode, used for local testing where tasks come only from the HTTP API) (client.go:37, stub.go).

The HTTPClient uses a 10s-timeout http.Client and authenticates every call with a Frappe token header — note this is token <key>:<secret>, different from the Bearer scheme the agent's own HTTP API uses (client.go:60).

ReportStatus/ReportProgress short-circuit to nil when jobName == "" — i.e. tasks submitted via the local HTTP API (which have no Frappe doc) skip reporting (client.go:86).

6. The poll loop in detail (internal/worker)

The worker is a single goroutine running two tickers (worker.go:47):

• poll ticker — fires every cfg.Agent.PollInterval (default 10s) → tick().
• reconcile ticker — fires every 30s → reconcileVMs().

Before the loop starts, two things happen once:

1. Crash recovery — ResetRunningTasks() flips any task left in running (because the agent died mid-task) back to pending so it gets retried (worker.go:50, queries.go:84).
2. Initial reconcile — reconcileVMs() runs immediately (worker.go:59).

One tick() (worker.go:128)

The four numbered phases in tick():

1. Poll & ingest — call frappe.Poll(serverName); for each returned FrappeTask, InsertTask into the local tasks table. Frappe is the source of truth for what to do; SQLite is the local durable queue (worker.go:130).
2. Claim one — ClaimPendingTask() (see locking below). If nil, return — nothing to do (worker.go:141).
3. Execute — w.execute(task) switches on task.Action and calls the matching handler, which drives the executor (worker.go:153, :180).
4. Report — on error: FailTask + ReportStatus("failed", …) (error truncated to the last 2000 chars to keep payloads small); on success: CompleteTask + ReportStatus("done") (worker.go:156).

Actions → handlers

Each handler calls w.progress(jobName, step) between phases, which forwards a human-readable step string to Frappe's update_progress (worker.go:201).

Idempotency & locking

• Single claim, atomic. ClaimPendingTask runs BEGIN → SELECT oldest pending (ORDER BY id ASC LIMIT 1) → UPDATE that row to running → COMMIT in one transaction (queries.go:32). Combined with SetMaxOpenConns(1) on the DB (all writes serialized through one connection, db.go:18), no task can be claimed twice.
• One task per tick. The worker processes exactly one task per 10s cycle, synchronously — execution blocks the loop until done (worker.go:140). Simple and predictable; the ceiling is throughput, not correctness.
• No automatic retry. FailTask marks the task failed immediately and does not increment retries or requeue (queries.go:76). (This contradicts architecture.md's "retries < 3" claim — the retry logic described there does not exist in the current code.) The only path back to pending is crash recovery on restart (ResetRunningTasks) — confirmed: FailTask is a one-shot terminal transition with no requeue.
• IP allocation is idempotent. AllocateIP returns the slug's existing IP if it already has one, so reset keeps the same external URLs (queries.go:157).

reconcileVMs() (the 30s loop)

Independent of the task queue. For every container the DB marks running, it calls exec.InspectState(slug); if the VM is actually down, it auto-restarts it via exec.Start(slug), and on restart failure records the real state so it doesn't thrash every 30s (worker.go:84). This is the self-healing that keeps boxes up across host hiccups.

Note: startLogCollector / stopLogCollector are intentional no-ops — Firecracker writes serial logs straight to flat files under /opt/devbox/logs/<slug>-serial.log, so nothing is collected into SQLite (worker.go:120).

7. The local HTTP API (internal/api)

A Gin server (release mode, Recovery + Logger middleware) on cfg.Agent.ListenAddr (default :8090) (api.go:43). It exists for operator/local debugging — in production Frappe submits work via polling, not this API, and the port is firewalled off the public internet.

Auth: every route except /health is wrapped in bearerAuth(authToken), which requires Authorization: Bearer <auth_token> to exactly match cfg.Agent.AuthToken; otherwise 401 (api.go:32). If auth_token is empty, the auth middleware is skipped entirely (api.go:54).

The golden-snapshot routes use a runtime type assertion to GoldenSnapshotBuilder; on the Docker backend (which doesn't implement it) the build routes return 400 and the status routes return {available:false} (api.go:213).

8. Config reference (config.yml)

Loaded by config.Load (config.go:63). The struct splits into four YAML sections. Defaults below are the in-code defaults applied before YAML is read; YAML values override defaults, and a few env vars override YAML (last column).

agent: (config.go:45)

frappe: (config.go:38)

firecracker: (config.go:17) — consumed by the Firecracker executor (pages 02/03/05)

docker: (config.go:54) — legacy/local backend only

config.example.yml ships only the frappe, agent, and docker sections — the Firecracker block is supplied per-server in the real config.yml.

9. Concurrency model

A small, deliberately simple set of goroutines (main.go):

• main goroutine — builds everything, then blocks on the signal channel.
• worker goroutine (go work.Run(ctx)) — the only thing that mutates VMs. It processes one task at a time, synchronously (execution blocks the poll loop), so there is no concurrency between tasks and no need for task-level locking beyond the atomic claim (worker.go:69).
• HTTP server goroutine (go httpServer.ListenAndServe()) — Gin spawns one goroutine per request. These goroutines only read DB rows and call read-style executor methods (Stats, WorkspaceTarStream), except POST /tasks which inserts a pending row for the worker to pick up.
• golden-snapshot builds spawn their own detached background goroutine so the HTTP handler returns 202 immediately (api.go:220).

Shared-state safety rests on SQLite, not Go mutexes: SetMaxOpenConns(1) serializes every write through a single connection, WAL mode allows concurrent reads, and a 5s busy_timeout waits out brief lock contention instead of failing with SQLITE_BUSY (db.go:11–30). There are no explicit mutexes or channels for task coordination in the agent layer — the single-connection DB + single-worker design makes them unnecessary.

10. The data model (internal/db)

Pure-Go SQLite via modernc.org/sqlite (no CGO → static binary). Schema created on Open via migrate() (db.go:40).

Task and Container are the Go structs the API serializes to JSON (models.go:5). Notable query helpers: ClaimPendingTask (atomic claim, §6), CompleteTask/FailTask (terminal transitions), ResetRunningTasks (crash recovery), UpsertContainer (INSERT…ON CONFLICT) (queries.go).

11. Deployment recap

Runs under systemd as the deploy user (devbox-agent.service):

• ExecStart=/opt/devbox/agent/devbox-agent -config /opt/devbox/agent/config.yml
• EnvironmentFile=-/opt/devbox/agent/agent.env — where the secret env overrides (AGENT_AUTH_TOKEN, FRAPPE_*) live.
• Restart=always, RestartSec=5 — self-healing process supervision.
• KillMode=process — only the agent dies on stop/restart, never the child Firecracker VMs (devbox-agent.service:17).
• Hardening: ProtectSystem=strict, ReadWritePaths=/opt/devbox, NoNewPrivileges=true.
• Logs to journald: journalctl -u devbox-agent -f.

Build & ship with ./scripts/deploy-agent.sh test|prod (see root CLAUDE.md).

Cross-references

• What exec.Create/Start/Reset actually do → pages 02/03 (Firecracker, rootfs/overlays) and 05 (snapshots).
• The Caddy route wiring triggered by ReconcileRoutes / ConfigureTLS → networking page.
• The Frappe side of pending_tasks / update_task → the control-plane page.