Agentic infrastructure
The question: What infrastructure do AI agents need to run long, stateful work against private systems, safely and cheaply?
Agents that debug real systems need a place to run: a shell, tools, files, and network access to private APIs, for hours at a time. This journey is about that place.
This page sums up the journey so far. The full write-ups are the entries below. Start with Entry 1: We Tried Every AI Sandbox. Then We Built Our Own. (12 min), or read the newest, Entry 2: Background Subagents in the Cloud: a Row, a Lease and a Fence Instead of a PID (17 min).
Entries (the full posts)
- We Tried Every AI Sandbox. Then We Built Our Own. 2026-03-21, start, 12 min, 4.0k tokens
Why hosted sandboxes did not fit, how we built ours on OpenSandbox and Kubernetes, and what running, paused and archived sandboxes cost. - Background Subagents in the Cloud: a Row, a Lease and a Fence Instead of a PID 2026-09-24, decision, 17 min, 4.9k tokens
How we run subagents detached in the cloud: all-or-none admission, only a run id on the broker, leases and fences, a charge that settles once, one wake-up turn, and what the agent itself needed.
What we think now
Revision 2, 2026-09-24. Older revisions: r1.
- Sandboxes must run inside our own network. Hosted sandboxes offer that only on enterprise plans, from about $3K a month. (entry 1)
- Persistence comes first. A sandbox has to survive timeouts and node loss; one 11-hour session survived a spot instance termination. (entry 1)
- Design for idle time. Per 20 GB, a running sandbox costs about $5 a month, a paused one about $1.60 (resumes in about 5 seconds), an archived one about $0.50 (about 30 seconds). At 200 sandboxes with 80% idle, that is about $200 a month instead of $1,000. (entry 1)
- Build on Kubernetes (OpenSandbox on EKS), with a network policy per sandbox and agents running unprivileged. (entry 1)
- Detached agent work is a row in Postgres, not a process. A lease says someone is working on it; a fence stops a worker that lost its claim from writing. (entry 2)
- Put only an id on the broker. Rebuild the agent's authority from current state on every attempt, so a replay can narrow it but never widen it. (entry 2)
- Write the terminal status, the completion and the charge in one transaction. An outbox delivers the completion and a recovery sweep settles the charge; receipts keyed by run id make a second charge a no-op. (entry 2)
- Background agents share the interactive worker pool, bounded by per-tenant caps, until p95 time to first token passes 2 seconds. (entry 2)
Open questions
- When does background agent load need its own worker pool, and how do we see it coming before chat slows down?
- Is container isolation enough, or do we need gVisor, Kata or Firecracker?
- Can the same sandbox run on a laptop?
- Can archiving and pre-warming be driven by usage numbers instead of fixed timers?
Follow this journey: RSS. Plain text version: /journeys/agentic-infrastructure.md