Task Orchestration End-to-End

This is the crown-jewel page. It traces what happens when a user clicks Create Dev Box — from the browser, through the Frappe control plane, across the poll gap, into the Go agent on a Hetzner server, down to Firecracker and Caddy, and all the way back to the dashboard turning green.

If you understand this one loop, you understand doppiobox. Everything else (billing, templates, the pool) hangs off it.

Read this alongside the actual code. architecture.md is stale; the file/line citations below are the source of truth.

1. The mental model: intent, not commands

The control plane (Frappe) and the data plane (the Go agent on each server) never call each other synchronously to do work. Frappe does not SSH in and run docker create. Instead:

1. A user action mutates a Dev Box document and records an intent as an Agent Job row (status Pending).
2. The Go agent, running on the target server, polls Frappe every ~10 seconds, pulls the jobs addressed to its server, and executes them locally.
3. The agent reports the outcome back to a Frappe endpoint, which updates the Dev Box status and pushes a realtime event to the dashboard.

So an Agent Job is a durable, queryable unit of work — a row in a table that says "server X should create box Y". The Dev Box doc holds the desired and observed state; the Agent Job holds the instruction to get there.

Why poll-based, not push?

This is the single most important design decision, so it gets its own list:

• Agents live on cheap dedicated Hetzner boxes behind NAT. They have no stable inbound hostname and no TLS cert of their own. Frappe cannot reliably open a connection to them. But they can always open an outbound HTTPS connection to Frappe. Polling inverts the dependency in the direction that actually works.
• The agent is the source of truth for what it can do. It claims work when it is free, at its own pace. Frappe never has to know whether the agent is busy, mid-reboot, or temporarily offline — pending jobs simply wait in the table.
• Crash resilience is free. If the agent dies mid-job, the work is still a row in Frappe (and in the agent's own SQLite). On restart the agent reconciles. Nothing is lost in an in-flight RPC.
• It's debuggable. Every intent is a row you can read in the Frappe desk (Agent Job list) with its status, action, payload, and error. No message bus to inspect.

The cost is latency: there is a poll-gap of up to one tick (~10s) between creating the job and the agent noticing it. For provisioning a dev environment that takes minutes, this is irrelevant.

2. The Agent Job doctype

benchspace/bench_space/doctype/agent_job/agent_job.json

The controller is intentionally empty — all the logic lives in the producers (Dev Box / template code) and the consumer (the agent + the callback endpoints):

# agent_job.py
class AgentJob(Document):
    pass

Fields

agent_job.json:42 lists the action options; agent_job.json:52 lists the status options.

Status state machine

Two important, non-obvious facts:

• The transition to Running happens inside the read endpoint, not via a separate claim call. When the agent polls, pending_tasks() flips each returned job to Running and commits before returning it (devbox.py:32-35,55-56). This guarantees a job is handed out at most once.
• There is no Failure → Pending retry edge on the Frappe side. A failed job is terminal in Frappe.

3. How Frappe exposes work to the agent

benchspace/api/devbox.py — three whitelisted endpoints form the entire agent-facing contract.

Authentication

The agent authenticates as a dedicated API user created per server (server.py:94 _create_agent_user) with System Manager role. The Go client sends a Frappe token header:

// client.go:60
req.Header.Set("Authorization", fmt.Sprintf("token %s:%s", c.cfg.APIKey, c.cfg.APISecret))

The api_key lives on the User; the api_secret is stored on the Server doc as agent_api_secret and injected into the agent's config via Ansible (server.py:172 _agent_variables).

3a. Poll — pending_tasks(server) GET

devbox.py:17

Request: GET /api/method/benchspace.api.devbox.pending_tasks?server=<server-name> (client.go:55).

Behaviour:

1. frappe.get_all("Agent Job", filters={"server": server, "status": "Pending"}, order_by="creation asc") — oldest-first, scoped to this server.
2. For each job: load it, set status = "Running", save. (This is the claim.)
3. Parse env_vars JSON; resolve devbox_template → its stable slug and attach as template_slug.
4. Commit, return a list.

Response shape (message is the Frappe envelope the Go client unwraps at client.go:72):

{
  "message": [
    {
      "name": "a1b2c3d4e5",
      "slug": "k9x2mq7p",
      "action": "create",
      "env_vars": { "GITHUB_TOKEN": "ghp_..." },
      "template_slug": "erpnext-v16"
    }
  ]
}

Maps to FrappeTask (client.go:15).

3b. Progress — update_progress(job, step) POST

devbox.py:61. Fire-and-forget sub-step reporting ("Creating container", "Waiting for site to be ready"). It does not change job status; it just frappe.publish_realtime("devbox_progress", ...) to the box owner so the dashboard can show a live spinner caption. The agent calls this liberally during long actions (see every w.progress(...) in worker.go).

3c. Result — update_task(job, status, error) POST

devbox.py:83. The terminal callback. status is "done" or "failed".

• Template builds (build_image, build_golden) branch to _handle_template_task_update (devbox.py:163) which updates the DevBox Template / Server Template Image records instead of a Dev Box.

• Otherwise: it sets the Agent Job to Success/Failure, then maps the action to the Dev Box status via DEVBOX_STATUS_MAP (devbox.py:6):

  action	on done → Dev Box status
  create	active
  start	active
  stop	stopped
  delete	deleted
  reset	active

  Any failed → Dev Box error with the error message.

• Side effects on success: a pool box auto-stops after create (devbox.py:121); a Provisioning subscription flips to Active (devbox.py:125). On failure of create/start, the subscription flips to Failed and a wallet refund is attempted (devbox.py:134-150).

• Finally frappe.publish_realtime("devbox_status_update", ...) to the owner so the dashboard re-renders.

4. The agent side: the worker loop

backbone/agent/internal/worker/worker.go

The worker runs two tickers (worker.go:61-65):

• poll ticker — fires every poll_interval (default 10s, config.go:66) → tick().
• reconcile ticker — fires every 30s → reconcileVMs() (covered in section 6).

On startup it does crash recovery first: db.ResetRunningTasks flips any locally-running SQLite task back to pending (worker.go:51), then an initial reconcileVMs().

tick() — worker.go:128

1. Poll Frappe        w.frappe.Poll(serverName)         → []FrappeTask
   Insert each into local SQLite   db.InsertTask(...)
2. Claim ONE           db.ClaimPendingTask(db)           → atomic sqlite tx, status->running
3. Execute             w.execute(task)                   → dispatch by action
4. Report              CompleteTask + ReportStatus("done")
                       or FailTask + ReportStatus("failed", errMsg)

Note the two-layer queue: Frappe's Agent Job table is the system-of-record; the agent mirrors each polled job into its own SQLite tasks table and claims/executes from there. This is what lets the agent survive a crash without re-asking Frappe (Frappe would no longer return the job, since it's already Running there).

Claiming — db.ClaimPendingTask (queries.go:32)

A single SQLite transaction: SELECT ... WHERE status='pending' ORDER BY id ASC LIMIT 1, then UPDATE ... SET status='running', then commit. One job per tick, oldest first. Serialized — the worker is single-threaded per tick.

Dispatch — w.execute (worker.go:180)

switch task.Action {
case "create":       return w.handleCreate(...)
case "start":        return w.handleStart(...)
case "stop":         return w.handleStop(...)
case "delete":       return w.handleDelete(...)
case "reset":        return w.handleReset(...)
case "build_golden": return w.handleBuildGolden(...)
case "build_image":  return w.handleBuildImage(...)
}

The handlers call the executor.Executor interface (executor.go:11), which has two implementations: docker.Executor and firecracker.Executor. The handler is backend-agnostic; the executor does the firecracker/docker work (covered in pages 02–03). After Create/Reset, the handler blocks on WaitForReady for port 8000 (Frappe site) and 8888 (code-server) before declaring success (worker.go:248-254).

Reporting back — frappe/client.go

• ReportStatus(jobName, status, errorMsg) → POST update_task (client.go:85). Errors are truncated to the last 2000 chars before sending (worker.go:162) to avoid oversized payloads.
• ReportProgress(jobName, step) → POST update_progress (client.go:122).

5. The master end-to-end sequence

One box creation, click to running:

The producer side of step 1–4 is dev_box.py:

# dev_box.py:42
def after_insert(self):
    self._send_agent_task("create")
    self.db_set("status", "creating")

_send_agent_task (dev_box.py:208) is the single chokepoint that turns any Dev Box action into an Agent Job. It also attaches the owner's GITHUB_TOKEN to env_vars for create/reset (dev_box.py:210-215).

6. Generalizing: every user action is the same loop

All the Dev Box action methods follow the identical pattern — optimistically set a transitional status, then enqueue a job — only the verb changes:

The matching agent handler for each action:

The two build_* actions are template-level, not box-level: they have no dev_box, and their result callback updates DevBox Template.build_status / Server Template Image.status instead of a box (devbox.py:163-228). A successful build_image auto-fans-out build_golden jobs to every server that should hold that template's snapshot (devbox.py:188,231 _trigger_golden_builds).

7. Error handling, retries, idempotency, stuck jobs

On failure

If w.execute returns an error (worker.go:156):

1. db.FailTask marks the local SQLite task failed immediately (queries.go:76).
2. ReportStatus(..., "failed", errMsg) posts to update_task, which sets the Agent Job Failure and the Dev Box error with the message, and refunds a failed subscription.

Retries

There is a retries column on the SQLite tasks table (models.go:14) and the worker logs attempt=task.Retries+1 (worker.go:150), but no code increments it or re-queues a failed task. FailTask is terminal. So in practice:

There is no automatic retry today. A failed action requires the user (or a System Manager) to re-trigger it from the dashboard, which inserts a fresh Agent Job.

Confirmed: FailTask (queries.go:76) is the only failure sink and it simply runs UPDATE tasks SET status='failed' — it does not increment retries or reset the task to pending. (The agent's own architecture.md describes a 3-attempt retry design, but that design was never implemented — another reason to treat that doc as stale.)

Timeouts

• The Go HTTP client to Frappe has a 10s timeout (client.go:51).
• WaitForReady bounds each create/reset readiness wait at 5 minutes per port (worker.go:248). Exceeding it returns an error → the job fails cleanly.
• Frappe → agent calls (fetch_containers, fetch_stats) use a 10s request timeout (dev_box.py:258).

Idempotency

The actions are not strictly idempotent, but the claim model makes double-execution very unlikely: a job leaves Pending the instant it's read (devbox.py:34), so a second poll never sees it. UpsertContainer in SQLite is upsert-by-slug, so re-running create/start on the same slug overwrites rather than duplicating the local record.

Stuck jobs (the known gap)

Consider: the agent polls, Frappe flips the job to Running, then the agent crashes before reporting a result.

• On the agent, ResetRunningTasks (worker.go:51, queries.go:84) flips its local task back to pending on restart, so it re-executes locally.
• But on Frappe, the Agent Job is already Running and pending_tasks only returns Pending jobs — so Frappe never re-offers it, and if the restarted agent's local copy also somehow lost the row, the job is orphaned in Running with the Dev Box stuck in creating.
• The safety net is the status-sync scheduler (next section): sync_all_dev_boxes reconciles a box stuck in creating against the agent's real container list and flips it to error if no container exists (dev_box.py:287-289).

There is no Frappe-side reaper that times out stale Running Agent Jobs. Recovery for the box status comes from the sync loop, not from the job table.

Confirmed against hooks.py:151-163: the registered scheduler_events are only sync_all_stats, replenish_pool, the four billing jobs, and the daily workspace cleanup — none of them touch the Agent Job table. Only the Dev Box status is reconciled (via _apply_status during sync).

8. The scheduler-driven reconcile loop

benchspace/hooks.py:151 registers these scheduler_events. The all bucket runs on every scheduler tick (~every 4 minutes by Frappe default — actual cadence is set by the framework's scheduler, not this app).

Note sync_all_dev_boxes (dev_box.py:363) exists and is the box-status reconciler, but it is not in the registered scheduler_events list — only sync_all_stats and replenish_pool are (confirmed in hooks.py:151-163). Status reconciliation therefore runs only on demand, via the per-box sync_status whitelisted method (dev_box.py:67), which calls _apply_status.

The reconcile loop (two independent reconcilers)

There are two reconcile loops at different layers — don't confuse them:

• Frappe's reconcilers (sync_all_*) pull observed state from the agent's HTTP API and write it onto Dev Box docs — they answer "what does the dashboard show?"
• The agent's reconcileVMs (worker.go:84, every 30s) is purely local: it checks that every container it thinks is running actually has a live VM process, and auto-restarts crashed VMs without telling Frappe — it answers "is the VM actually up?" This is why a box can silently recover from a host hiccup.

9. Cheat sheet

• Producer of work: dev_box.py:208 _send_agent_task (and _trigger_golden_builds for templates) → inserts Agent Job {status: Pending}.
• Agent-facing API: pending_tasks (poll+claim), update_progress (live steps), update_task (terminal result) — all in api/devbox.py.
• Auth: Frappe token <api_key>:<api_secret>, per-server agent user (server.py:94).
• Poll cadence: 10s (config.go:66); agent VM reconcile: 30s (worker.go:64).
• Claim semantics: job leaves Pending on read in Frappe; agent independently claims from its own SQLite (queries.go:32). At-most-once handoff.
• Status mapping on done: DEVBOX_STATUS_MAP (devbox.py:6).
• No automatic retries; no Frappe-side stuck-job reaper. Box-status recovery comes from _apply_status during sync.