Deployment, Ansible & Server Provisioning

This page is about getting a bare Hetzner server to the point where it can run boxes — and keeping the software on it up to date. Two very different tools do two very different jobs here, and the whole page hinges on telling them apart:

• Ansible provisions the host — installs Firecracker, Caddy, the bridge network, the kernel, the agent itself. This is occasional, server-level, one-shot-ish work.
• The Go devbox-agent runs the boxes — creating, starting, stopping, snapshotting VMs all day long. This is continuous, per-box runtime work (covered in pages 02–05).
• Frappe is the control plane that orchestrates both: it triggers Ansible runs to provision a server, and it queues tasks the agent polls to manage boxes.

Audience: a newcomer who may never have touched Ansible. We teach it from first principles, then tie every concept to a file in this repo.

All Ansible lives under backbone/ansible/. The Python that drives Ansible from inside Frappe is benchspace/runner.py.

1. What Ansible is (first principles)

If you have never used Ansible, read this section before anything else — the rest of the page assumes it.

Ansible is an agentless IT-automation tool. You describe the desired end state of a server in YAML, point Ansible at the server, and it makes the server match. "Agentless" means there is nothing to pre-install on the target: Ansible just SSHes in and runs commands. (Contrast with tools like Puppet/Chef that need a daemon running on every managed host. The only requirement on the target is SSH + Python — which is why setup-from-rescue.sh installs python3 before running anything; see §4.)

Imperative shell script vs. declarative playbook

A shell script is imperative — a list of commands in order. Run it twice and it may break (it re-creates a user that already exists, re-appends a line, etc.). An Ansible playbook is declarative — you state what should be true, and Ansible figures out whether it already is.

# Declarative: "this directory should exist, owned by deploy". Safe to run 100 times.
- name: Create /opt/devbox/agent directory
  file:
    path: /opt/devbox/agent
    state: directory
    owner: "{{ deploy_user }}"

This re-runnability is called idempotency: running a playbook again only changes what has drifted from the desired state; everything already correct is reported "ok" and left alone. The stats at the end of a run (ok, changed, skipped, failures) come straight from this model — and doppiobox stores them on the Ansible Play doctype (§6).

The vocabulary (all of it appears in this repo)

A role is just an organising convention: roles/<name>/tasks/main.yml is the task list, roles/<name>/templates/*.j2 are Jinja2 templates rendered with variables, roles/<name>/handlers/main.yml are the notify-only tasks. Variables shared across everything live in group_vars/all.yml (versions, package lists, the deploy user name).

ansible.cfg sets the defaults for the whole directory:

[defaults]
inventory = inventory.ini
host_key_checking = false
any_errors_fatal = true

any_errors_fatal = true means one failed task aborts the whole run — important, because half-provisioned servers are worse than un-provisioned ones.

2. The roles in this repo

The full provisioning playbook (playbook.yml) composes these roles in order. Each is a focused, idempotent bundle.

The agent role is where the two worlds meet: its config template (roles/agent/templates/devbox-agent-config.yml.j2) is what tells the Go agent which backend to use, which Frappe URL to poll, and the auth token. Ansible writes the agent's config; the agent then runs on its own.

3. The playbooks: which one does what

There are several playbooks because provisioning a fresh server, re-deploying just the agent, and rebuilding images are different-sized jobs.

Tags (used by deploy.yml) let one playbook do sub-jobs: --tags X runs only tasks tagged X; --skip-tags golden runs everything except golden. That is why "rebuild images but skip the snapshot" is --tags images --skip-tags golden.

4. The provisioning journey: rescue mode → running stack

A brand-new Hetzner dedicated server (must be bare metal — Firecracker needs /dev/kvm, cloud VMs do not have it) goes through this pipeline. The automated path is setup-from-rescue.sh; the manual steps are in SERVER_SETUP.md.

Step by step (from setup-from-rescue.sh):

1. Rescue mode — In the Hetzner Robot panel, activate the rescue system and do a hardware reset. The server boots a tiny in-memory Linux you can SSH into. The script sanity-checks this by grepping /etc/motd for "rescue".
2. installimage — installimage.conf is uploaded and installimage -a -c is run. This is Hetzner's OS installer; the config declares the disk layout: software RAID1 across both NVMe drives, /boot (1 GB ext4) on md0, and the root filesystem (450 GB ext4) on LVM (md1 → vg0 → root), from an Ubuntu 24.04 base image.
3. Reboot into the freshly installed OS; the script waits up to 5 min for SSH to come back and removes the stale host key.
4. Install Python3 — fresh Ubuntu may lack it; Ansible needs Python on the target to run its modules.
5. Run Ansible — deploy-test.yml installs the entire stack (Docker, Firecracker + jailer + vmlinux, the br-devbox bridge with NAT, Caddy with on-demand TLS, the directory tree, the rootfs image, and the agent service), then builds the golden snapshot.

SERVER_SETUP.md also documents a 12-point verification checklist (OS version, RAID/LVM layout, /dev/kvm, Docker, Firecracker binaries, bridge, NAT, Caddy, agent /health, rootfs image, golden status, and creating a test VM).

5. The deploy-agent.sh workflow (the everyday path)

Once a server is provisioned, you rarely re-run the full playbook. The common operation is "I changed the Go agent, push it." scripts/deploy-agent.sh is the fast, no-Ansible path — it builds the binary locally and SSHes the artifacts up directly.

What it always does, in order:

1. Loads .env and picks the target (test is the default; prod switches to the HETZNER_DEDICATED_SERVER_* credentials).
2. Syncs the Dockerfile to /opt/devbox/images/Dockerfile (always — build_image on the server needs it).
3. Builds the agent: CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build (static Linux amd64 binary, cross-compiled from macOS).
4. Stops devbox-agent via systemd, uploads the binary to /opt/devbox/agent/devbox-agent, restarts, and prints systemctl is-active.

The flags:

Two ways to deploy the agent. deploy-agent.sh is the manual operator path (build locally, scp up). The Frappe deploy_agent button runs the deploy-agent.yml Ansible playbook instead (§6). Same outcome, different driver — the script for human ops, Ansible for control-plane-initiated ops.

6. How Frappe drives Ansible — Ansible Play & Ansible Task

This is the "Go agent + Ansible cooperate" angle made concrete. doppiobox does not shell out to the ansible-playbook CLI. It runs Ansible in-process via its Python API and streams progress into Frappe doctypes in real time. The glue is the Ansible class in benchspace/runner.py.

The Server doctype: registering a host

A Hetzner server is modelled as a Server doc.

Two whitelisted actions drive provisioning:

• prepare_server() — guards on status, generates the agent token, creates the agent's callback user (_create_agent_user), sets status Preparing, and enqueues run_prepare_server on the long queue (30-min job). That background job builds an Ansible(server, playbook="playbook.yml", ...) and calls .run(). On success → status Active; on failure → Error with the play's error captured.
• deploy_agent() — only on Active servers; enqueues run_deploy_agent, which runs the deploy-agent.yml playbook the same way.

The variables passed to Ansible are assembled in _agent_variables(): the Frappe URL, the agent token, the callback user's API key/secret, the server name, and the backend. These land in the agent's config.yml via the role template — closing the loop so the freshly-deployed agent knows how to poll this exact Frappe instance.

Ansible Play / Ansible Task: modelling a run

When Ansible.run() is called, create_ansible_play() first inserts one Ansible Play doc (status Pending), then parses the playbook with Playbook.load(...) and pre-creates one Ansible Task child-ish doc per task it finds (skipping meta tasks), recording task name + role.

The Play doc holds the run-level record: playbook, server (Link), status (Pending/Running/Success/Failure), timing (start/end/duration), the variables JSON, the per-run stat counters (ok, failures, changed, unreachable, skipped, rescued, ignored), and an aggregated error. Each Ansible Task carries its own status (adds Skipped/Unreachable), timing, and captured output/error/exception/result.

Progress is wired through a custom Ansible callback (AnsibleCallback, subclass of CallbackBase). Ansible invokes its v2_* hooks as the run proceeds, and each hook writes back to the docs:

Because these callbacks run inside a long background job, DB connections can drop — reconnect_on_failure() wraps every hook to reconnect and retry once. After every update the callback calls publish_play_progress(), which frappe.publish_realtime("ansible_play_update", ...) so a desk UI can show a live progress bar. (The Ansible Task doctype also re-publishes on on_update.) When a Play is trashed, its child Tasks are deleted in Ansible Play.on_trash.

The division of labour

The clean split:

• Ansible = host shape. Runs occasionally (provision a new server, push a new agent build, rebuild images). Idempotent. Driven by the Server doc's two buttons.
• Go agent = box lifecycle. Runs continuously, polls Frappe every ~10s for box tasks, owns VM create/start/stop/delete/snapshot. (Pages 02–05.)
• Frappe = orchestration. Triggers Ansible for server-level work, queues tasks for the agent for box-level work, and records both (Ansible Play/Task docs; box/task docs).

7. Golden snapshot & rootfs rebuild as deploy operations

Page 03 explains what the golden snapshot is (a frozen, Frappe-ready VM that new boxes clone in ~3–20s). Here we cover the ops triggers — the deploy-time side.

Rootfs rebuild — the base ext4 image (/opt/devbox/images/frappe-v16.ext4) is what every VM boots from. It is rebuilt by build-rootfs.sh on the server (the docker_images role / deploy.yml --tags images / deploy-agent.sh --rebuild-rootfs). Takes ~20 min; run it whenever the Dockerfile or base image inputs change. After a rootfs rebuild, the golden snapshot is stale and must be rebuilt.

Golden snapshot build — triggered over the agent's HTTP API, not via SSH commands:

AUTH="Authorization: Bearer $(grep auth_token /opt/devbox/agent/config.yml | awk '{print $2}' | tr -d '\"')"
curl -X POST -H "$AUTH" http://localhost:8090/golden-snapshot/build   # ~10 min
curl       -H "$AUTH" http://localhost:8090/golden-snapshot/status    # poll until available

The Ansible playbooks automate exactly this: deploy.yml and deploy-test.yml both grep the token out of config.yml, POST to /golden-snapshot/build, then poll /golden-snapshot/status with until: golden_status.json.available (retries 40 × 15s ≈ 10 min). This is a neat example of Ansible and the agent cooperating: Ansible can't build a snapshot itself, so it asks the agent to and waits.

8. Scale testing

backbone/scripts/scale-test.sh is an operator script (not Ansible) that hammers the agent API on a provisioned server to validate it under load: it concurrently creates N VMs from the golden snapshot, checks each gets a unique IP and answers on ports 8000 (site) + 8080 (code-server), measures memory (~2 GB/VM), then stops/starts/deletes all of them and verifies cleanup (no leftover overlays, volumes, TAP devices, or IP allocations in the agent's SQLite DB). Use it after provisioning a new server or rebuilding the golden snapshot to confirm the host is healthy end-to-end.

9. The VS Code extension (brief)

backbone/extensions/benchspace-devbox/ is a small VS Code / code-server extension (benchspace-devbox, v0.3.0) that ships inside the box image, not on the host. Its .vsix is copied into the rootfs by the docker_images role (benchspace-devbox-0.1.0.vsix in the base image inputs) so it's pre-installed in every box's code-server. It contributes box-side conveniences (e.g. a BenchSpace: View Site command). It is part of the box runtime image, orthogonal to host provisioning — mentioned here only because the deploy pipeline bundles it.

Cross-references

• Page 02 — what Firecracker is (the microVM the firecracker role installs).
• Page 03 — what the golden snapshot is (this page covers the ops trigger).
• Page 05 — Caddy reverse proxy + on-demand TLS (the caddy role).
• CLAUDE.md (repo root) — the operational cheat-sheet: server IPs, key paths, agent API, deploy-agent.sh usage.