Background Subagents in the Cloud: a Row, a Lease and a Fence Instead of a PID
On this page
- Why the cloud is a different problem
- The life of one detached run
- Admission: all or none
- Only the run id crosses the broker
- A lease for liveness, a fence for safety
- Nobody gets billed twice
- One wake-up, however many children
- Statuses that say who acts next
- The agent half
- What the review loop found
- The open tradeoff
- What we do now
Building it took one week. Making it survive crashes, replays and double billing took three.
Background subagents let our main agent, or a specialist agent, hand bounded work to child agents and end its turn at once. The children keep running in the cloud. When they finish, the parent wakes up and answers the user with what they found.
Claude Code and Codex already have this feature on a laptop. But a laptop harness gets a lot for free from the operating system. There is a process to wait on, a local disk to write to, and a signal when a worker dies. Our children run on a shared pool of cloud workers. None of those primitives exist there.
Writing the feature was the small part. Most of the time went into proving three things:
- A detached run survives a worker that dies mid-step.
- A replayed message cannot widen what a child is allowed to do.
- Nobody gets billed twice, and nobody gets billed zero times.
Why the cloud is a different problem
On a laptop, the harness starts a child process and keeps its PID. The PID answers every question: is the child alive, who owns the work, where is its state, did it finish. In the cloud, a Celery worker picks up a message, and any worker can die at any moment. The broker may deliver the same message again. The parent’s own turn ended minutes ago.
So every primitive needs a replacement that lives in the database.
The distributed-systems half of this was familiar work in an unfamiliar place: leases, fences, outboxes and idempotent delivery. The agent half comes at the end.
The life of one detached run
Three words in this post have exact meanings:
- A subagent, or child, is one piece of delegated work. It has a stable handle for its whole life.
- A run is one execution of a child, with its own run id and its own row in Postgres.
- A continuation is a follow-up task for a finished child. It starts a new run under the same handle, and the old run becomes
superseded.
A parent calls spawn_subagents with execution_mode: background. The prompt suggests 2 to 4 children and allows up to 12. The call returns one handle per child, and the parent’s turn ends.
The rest of this post walks through those steps, one failure at a time.
Admission: all or none
A fan-out asks one question. If we admit half of it, the parent gets half an answer and does not know it. So admission is all or none.
- A cheap preflight counts active children without a lock. Most refusals stop here.
- Then one transaction takes a global admission lock, counts again, and reads the credit balance inside the lock. Two fan-outs that arrive together cannot both see room that only one of them fits in.
- A balance we cannot read counts as empty. The gate fails closed.
- Each child gets its own run row, keyed by the id of the spawn call and the child index. If the same call is delivered twice, it finds its own rows and admits nothing new.
- The caps are 15 active children per workspace and 30 per organization. They count
queuedandrunningchildren only. A child that waits for a person has released its worker, so it does not count. When the person answers, the child passes the same caps again before a worker takes it.
A refusal goes back to the model with a named reason. That matters more than we expected; see the agent half.
Only the run id crosses the broker
This is the task, trimmed to what matters:
@celery_app.task(
name="background_subagent.run",
bind=True,
acks_late=True,
time_limit=1800,
soft_time_limit=1740,
)
async def run_background_subagent(self, run_id: str) -> None: ...
Three choices are in those lines.
- The argument is a run id and nothing else. The worker loads the model, the tools, the connections and the budget from Postgres. A replayed or forged message can only point at a row that already exists.
acks_late=Trueand no retry. A worker that dies leaves its message unacknowledged, so the broker can deliver it again later. Celery itself never retries the task, because a retry would re-enter a child that may already have done something. The reaper recovers the child instead, and a late copy of the message has to win the same claim as any other worker.- The 30-minute limit is far below the broker’s one-hour visibility timeout. With late acknowledgement, a task that outlives the visibility timeout is delivered to a second worker while the first one still runs it. The lease and the fence would keep the second worker out, but the limit stops the case from happening at all.
Rehydration trusts nothing but the id
The run row holds a versioned recipe for the child. The worker rebuilds the child from it, and checks it against the workspace as it is now.
- An unknown recipe version, or an unknown field, fails. Nothing is filled in with a default.
- A bring-your-own-key model is authorized again against the workspace.
- The child’s connections are the recipe’s connections intersected with the ones the spawning agent has now.
The last rule came from the review loop. The first version rebuilt connections from the recipe alone. A user could remove a connection from an agent, and a child that was recovered after that would get the connection back. Nothing about it looked wrong in a single run. It needed a worker to die at the right moment.
A lease for liveness, a fence for safety
A worker claims a child with one atomic UPDATE ... RETURNING. The claim sets a lease that expires in 30 seconds and increments a fence token. While the child runs, a heartbeat renews the lease every 10 seconds. The renewal must match both the owner and the fence.
The two do different jobs:
- The lease is for liveness. It answers “is anybody still working on this?” When it expires, another worker may take the child.
- The fence is for safety. A worker that lost its claim is not always dead. It may be stalled, or cut off from the database for a minute. When it comes back, it still believes it owns the child. Its next heartbeat fails because the fence moved, and the worker cancels its own execution. A write that it makes before then to the run row, the tool-call ledger or an approval carries the old fence, and the database rejects it.
A lease without a fence lets two workers write to one child. A fence without a lease never lets anyone take over.
The reaper
A reaper runs every 2 minutes on a low-priority queue. It finds children with no live lease and re-sends the same run id. The new worker loads the child’s LangGraph checkpoint from Postgres, so the child continues from its last step. It does not start again.
- A
queuedchild gets 10 minutes of grace first, because a busy pool delays a message without losing it. - A child gets 3 recovery attempts. After that it fails with
RECOVERY_EXHAUSTED, and the parent hears about it. A child that retries forever looks like live work that never moves. - The reaper reserves an attempt with an atomic
UPDATEbefore it sends anything. Two reapers that scan the same child can each spend an attempt, but the count never passes 3. - A worker that is shutting down does not start a child. It hands the run back, and another worker takes it.
Before it claims, a worker also skips a child that is already terminal, has a stop requested, or waits for a person to resolve a tool call. A child that waits for approval is different: Approve sends its run id again, and a worker claims it then.
Nobody gets billed twice
A child’s charge is written in the same transaction that makes the child terminal. That transaction writes three things: the status, the completion outbox row and the parked charge.
After the commit, the charge goes to the ordinary credit task. That task writes the quota change, the usage log and a receipt keyed by the child’s run id, in one transaction. A second delivery of the same charge finds the receipt and changes nothing.
If the publish is lost, the charge stays parked on the run row. A credit recovery sweep runs every 10 minutes and publishes it again. The work is done and the user has the answer, so the sweep only has to settle the money.
The review loop found a gap in that sweep. The terminal path claims the charge, commits, and then publishes. In the first version, a crash between the commit and the publish left a charge that was claimed and never billed. The sweep skipped it, because the claim said someone was on it. Now the sweep treats an old claim as abandoned and publishes the charge again. The receipt makes that safe even when the first publish did arrive.
One wake-up, however many children
The parent’s stream ended long ago. A finished child cannot just reply into it.
- The terminal transaction writes a completion outbox row. After the commit, the child adds its id to the conversation’s list of pending completions.
- Then it checks whether a turn is already streaming. It checks; it does not take the lock. If no turn is streaming, it starts one.
- A running turn drains the whole list on every model step. So a child that finishes during a turn is folded into it. Five children that finish close together give one answer, not five.
- The drain removes the ids, but that is not proof that a user saw them. The outbox row is retired only when the turn that showed the completion has ended. A turn that dies before it answers leaves the completion owed, not lost.
- An outbox sweep every minute re-sends any wake-up whose first dispatch was lost.
The wake-up carries an id, not content. Each drain reads the findings from Postgres. They go to the parent as untrusted data, below the lines the parent may act on, and the parent is told not to follow instructions it finds there. That lowers the risk of a prompt injection through a child. It does not remove it.
Statuses that say who acts next
A background child can outlive its parent’s turn by hours. So a status must say who acts next: a worker, a person, or nobody.
waiting_for_approval: our approval policy stopped a tool call and asked a person. The parent stream is gone, so approval is its own endpoint and not a chat reply. Approve sends the same run id again, and the child passes the caps again. Reject cancels the child.waiting_for_resolution: a tool may have changed something, and nothing can prove whether it did. The child does not guess, and it does not call the tool again, because a second call is the exact failure this state exists to prevent. A person states what the tool really did. Then the child passes the caps again and resumes with that answer, or the person closes it.cancel_requested: the Stop button and thecancel_subagentstool share one policy. Aqueuedchild, or one that waits for approval, is cancelled at once. A child that waits for resolution cannot be stopped until a person resolves it. A child that a worker holds gets the request, and the worker writescancelledat the next step. That way one writer, and only one, makes the child terminal.superseded: a continuation started a new run under the same handle. The old run is kept for the record.
Both waiting statuses release the worker. A child can wait for a person for days and hold no thread.
The agent half
The distributed-systems work was what we expected. The surprise was the agent. An agent that can delegate needs three things. Without each one, the model does something reasonable that does not help the user.
A refusal it can act on
A bare error such as “capacity exceeded” gives the model two moves. It retries the same call, or it asks the user what to do next. Every refusal now has a code, the context, and the next action, and it says who decides:
BACKGROUND_SUBAGENTS_DISABLED: new background subagent work is
unavailable.
Context: requested execution_mode=background; no new background
child was admitted; existing runs are unchanged.
NEXT_ACTIONS (do NOT ask the user): use foreground only when
waiting in this turn still satisfies the request; otherwise
tell the user that background execution is unavailable and
continue with work this turn can complete.
Some refusals do need the user. A follow-up message to a child that the user stopped is one: only the user can start it again. Those refusals leave out the “do NOT ask the user” prefix. A refusal that says both “do not ask the user” and “the user must decide” makes the model guess.
An identity it can still hold after compaction
A handle that lives only in a tool result does not survive compaction. The summary keeps the story and drops the ids. The parent then knows it started an investigation, but it cannot reach the child that runs it.
So the handle lives in Postgres. list_subagents returns every child’s handle, alias, status and task for the conversation. The model holds the handle, not a run id, so a continuation does not break its reference.
At first, our loop guard, which stopped an agent that repeated the same tool call, blocked list_subagents. Listing again was the correct recovery move, so we exempted it. We have since removed the loop guard.
A reason to end its turn
Give a model a list of running children, and it waits. It polls the status, or it keeps the turn open. The tool description says the opposite, in the same words each time:
execution_mode: background (default) returns handles now,
children run past this turn and each completion wakes you;
state what started, do not poll.
list_subagents repeats it: “Never poll; background completion wakes you.” There is no wait tool. The instruction only works because the outbox makes it true. Ending the turn is safe when a completion can never be lost.
What the review loop found
The review ran on CloudThinker Code Review. Two of the bugs it found are above: the revoked connection and the stranded charge. The third reported a child as finished while the write that parked its outcome had rolled back. The parent would never have been woken for it. Now the status and the outcome commit in one transaction, or neither does.
None of these bugs shows up when every worker lives. Each one needs a crash, a redelivery or a revoke at one exact moment, so tests that only follow the happy path pass all of them. The fixes came with regression tests that force the moment: a stop that races a claim, a publisher that crashes after the commit, a heartbeat lost at the terminal write.
The open tradeoff
Background children share the worker pool with interactive chat. There is no dedicated queue and no reserved share.
The admission caps bound background load for each tenant. Nothing reserves a share of the pool for chat, so we watch chat latency to see when the caps stop being enough.
A dedicated queue would split the capacity in a fixed way. Background work would wait while chat threads sat idle, and the other way around. It would also need its own scaling for a feature whose steady volume we do not know yet. So we deferred it, and wrote down when to change our minds: a measured p95 time to first token above 2 seconds in interactive chat, caused by background load.
What we do now
- A background child is a row in Postgres, not a process. The row owns its status, its lease, its fence and its checkpoint.
- Only the run id goes on the broker. Every attempt rebuilds the child’s authority from current state, so a replay can narrow it but never widen it.
- The terminal status, the completion and the charge are written in one transaction. An outbox delivers the completion, and a recovery sweep settles a parked charge. Receipts keyed by run id make a second charge a no-op.
- The parent answers once per wake-up, however many children finished.
- Every refusal the model sees names the next action. Every handle the model needs can be listed again after compaction. The model is told to end its turn, and the outbox makes that safe.
- Background load shares the interactive pool until time to first token says otherwise.
<< Previous: We Tried Every AI Sandbox. Then We Built Our Own. | Up: all entries