CloudThinker Engineering

Background Subagents in the Cloud: a Row, a Lease and a Fence Instead of a PID

On this page

Building it took one week. Making it survive crashes, replays and double billing took three.

Background subagents let our main agent, or a specialist agent, hand bounded work to child agents and end its turn at once. The children keep running in the cloud. When they finish, the parent wakes up and answers the user with what they found.

Claude Code and Codex already have this feature on a laptop. But a laptop harness gets a lot for free from the operating system. There is a process to wait on, a local disk to write to, and a signal when a worker dies. Our children run on a shared pool of cloud workers. None of those primitives exist there.

Writing the feature was the small part. Most of the time went into proving three things:

  1. A detached run survives a worker that dies mid-step.
  2. A replayed message cannot widen what a child is allowed to do.
  3. Nobody gets billed twice, and nobody gets billed zero times.

Why the cloud is a different problem

On a laptop, the harness starts a child process and keeps its PID. The PID answers every question: is the child alive, who owns the work, where is its state, did it finish. In the cloud, a Celery worker picks up a message, and any worker can die at any moment. The broker may deliver the same message again. The parent’s own turn ended minutes ago.

So every primitive needs a replacement that lives in the database.

Each laptop primitive and its cloud replacement: a PID becomes a run row, a live process becomes a lease, process ownership becomes a fence token, SIGTERM and restart becomes the reaper, wait() becomes an outbox row, and local memory becomes a Postgres checkpointOn a laptopIn the clouda PID to wait ona run row in Postgresstatus, owner, lease, fencethe process is alivea lease30 s, renewed every 10 sone process owns the worka fence token+1 on every claimSIGTERM, then restartthe reaper re-sends the run idthe checkpoint continueswait() returnsan outbox rowwritten with the terminal statusstate in local memorya Postgres checkpointper child thread

The distributed-systems half of this was familiar work in an unfamiliar place: leases, fences, outboxes and idempotent delivery. The agent half comes at the end.

The life of one detached run

Three words in this post have exact meanings:

  • A subagent, or child, is one piece of delegated work. It has a stable handle for its whole life.
  • A run is one execution of a child, with its own run id and its own row in Postgres.
  • A continuation is a follow-up task for a finished child. It starts a new run under the same handle, and the old run becomes superseded.

A parent calls spawn_subagents with execution_mode: background. The prompt suggests 2 to 4 children and allows up to 12. The call returns one handle per child, and the parent’s turn ends.

The eight steps of one background child, from the parent call through admission, the broker, the claim, rehydration, the run, the terminal transaction and the wake-up, with the reaper sending the run id through the broker again when a worker dies1. Parent calls spawn_subagentsbackground: handles return, the turn ends2. Admission, all or nonelock, caps, credits, one row per child3. Brokerthe message carries only the run id4. Claima 30 s lease and a new fence token5. Rehydrate from Postgresmodel, BYOK key, tools, connections, budget6. Run the child graphcheckpoint every step, heartbeat every 10 s7. One terminal transactionstatus + completion outbox row + parked charge8. Wake the parentone turn reports every child that finishedNo roomrefuse the whole fan-outWorker dies mid-stepthe lease is not renewedReaper, every 2 minre-sends the same run idthrough the brokerlease expires

The rest of this post walks through those steps, one failure at a time.

Admission: all or none

A fan-out asks one question. If we admit half of it, the parent gets half an answer and does not know it. So admission is all or none.

Admission runs an unlocked preflight, then takes the global lock, recounts both caps, reads credits inside the lock, inserts one run row per child and commits; any failed check refuses all N childrenParent asks for N childrenone spawn_subagents callPreflight, no lockcount active children: workspace, then orgTake the global admission lockheld until the transaction commitsRecount both caps under the lockanother fan-out may have landedRead credits inside the locka balance we cannot read counts as emptyInsert N run rows, one per childkeyed by the spawn call id and the child indexCommit, then enqueue N run idsthe lock is released by the commitRefuse all Nworkspace, org or creditthe model gets the reasonCaps count queued and running children only: 15 per workspace, 30 per organization.No partial admit: half a fan-out answers half the question.
  • A cheap preflight counts active children without a lock. Most refusals stop here.
  • Then one transaction takes a global admission lock, counts again, and reads the credit balance inside the lock. Two fan-outs that arrive together cannot both see room that only one of them fits in.
  • A balance we cannot read counts as empty. The gate fails closed.
  • Each child gets its own run row, keyed by the id of the spawn call and the child index. If the same call is delivered twice, it finds its own rows and admits nothing new.
  • The caps are 15 active children per workspace and 30 per organization. They count queued and running children only. A child that waits for a person has released its worker, so it does not count. When the person answers, the child passes the same caps again before a worker takes it.

A refusal goes back to the model with a named reason. That matters more than we expected; see the agent half.

Only the run id crosses the broker

This is the task, trimmed to what matters:

@celery_app.task(
    name="background_subagent.run",
    bind=True,
    acks_late=True,
    time_limit=1800,
    soft_time_limit=1740,
)
async def run_background_subagent(self, run_id: str) -> None: ...

Three choices are in those lines.

  • The argument is a run id and nothing else. The worker loads the model, the tools, the connections and the budget from Postgres. A replayed or forged message can only point at a row that already exists.
  • acks_late=True and no retry. A worker that dies leaves its message unacknowledged, so the broker can deliver it again later. Celery itself never retries the task, because a retry would re-enter a child that may already have done something. The reaper recovers the child instead, and a late copy of the message has to win the same claim as any other worker.
  • The 30-minute limit is far below the broker’s one-hour visibility timeout. With late acknowledgement, a task that outlives the visibility timeout is delivered to a second worker while the first one still runs it. The lease and the fence would keep the second worker out, but the limit stops the case from happening at all.

Rehydration trusts nothing but the id

The run row holds a versioned recipe for the child. The worker rebuilds the child from it, and checks it against the workspace as it is now.

The broker message carries only the run id; the worker reads the recipe from the run row and the current workspace state, fails closed on unknown fields, re-authorizes the BYOK key, intersects connections with the agent’s current ones, and the child can do what it was granted or lessBroker message{ run_id }Run rowversioned recipe: model,BYOK key, tools, connections,budgetWorkspace todayBYOK authorization,the agent's connectionsRehydratethe worker trusts nothing but the idUnknown version or fieldfail closed, no defaultsBYOK key re-authorizedagainst the workspace nowConnections = recipe ∩ currenta revoked one does not come backThe child may do what it was grantedor less, never more
  • An unknown recipe version, or an unknown field, fails. Nothing is filled in with a default.
  • A bring-your-own-key model is authorized again against the workspace.
  • The child’s connections are the recipe’s connections intersected with the ones the spawning agent has now.

The last rule came from the review loop. The first version rebuilt connections from the recipe alone. A user could remove a connection from an agent, and a child that was recovered after that would get the connection back. Nothing about it looked wrong in a single run. It needed a worker to die at the right moment.

A lease for liveness, a fence for safety

A worker claims a child with one atomic UPDATE ... RETURNING. The claim sets a lease that expires in 30 seconds and increments a fence token. While the child runs, a heartbeat renews the lease every 10 seconds. The renewal must match both the owner and the fence.

Worker A claims with fence 7 and stalls; the lease expires; the reaper re-sends the run id; worker B claims with fence 8 and continues from the checkpoint; worker A wakes, writes its status with fence 7, is rejected, and its failed heartbeat makes it cancel itselftimeWorker ARun rowReaperWorker Bclaim: fence 7, lease 30 sheartbeat renews the leasestalls mid-steplease expiresno live leasere-send run idclaim: fence 8continues fromits checkpointwakes up, writesstatus with fence 7stale fence: rejectedits heartbeat fails too,so A cancels itself

The two do different jobs:

  • The lease is for liveness. It answers “is anybody still working on this?” When it expires, another worker may take the child.
  • The fence is for safety. A worker that lost its claim is not always dead. It may be stalled, or cut off from the database for a minute. When it comes back, it still believes it owns the child. Its next heartbeat fails because the fence moved, and the worker cancels its own execution. A write that it makes before then to the run row, the tool-call ledger or an approval carries the old fence, and the database rejects it.

A lease without a fence lets two workers write to one child. A fence without a lease never lets anyone take over.

The reaper

A reaper runs every 2 minutes on a low-priority queue. It finds children with no live lease and re-sends the same run id. The new worker loads the child’s LangGraph checkpoint from Postgres, so the child continues from its last step. It does not start again.

  • A queued child gets 10 minutes of grace first, because a busy pool delays a message without losing it.
  • A child gets 3 recovery attempts. After that it fails with RECOVERY_EXHAUSTED, and the parent hears about it. A child that retries forever looks like live work that never moves.
  • The reaper reserves an attempt with an atomic UPDATE before it sends anything. Two reapers that scan the same child can each spend an attempt, but the count never passes 3.
  • A worker that is shutting down does not start a child. It hands the run back, and another worker takes it.

Before it claims, a worker also skips a child that is already terminal, has a stop requested, or waits for a person to resolve a tool call. A child that waits for approval is different: Approve sends its run id again, and a worker claims it then.

Nobody gets billed twice

A child’s charge is written in the same transaction that makes the child terminal. That transaction writes three things: the status, the completion outbox row and the parked charge.

One terminal transaction writes the status, the completion outbox row and the parked charge; after commit the charge is published to the credit task; a lost publish is recovered by credit recovery every 10 minutes, which reclaims the stale claim and re-publishes; a second delivery finds the receipt and charges nothingOne terminal transactionstatus = succeededcompletion outbox rowparked credit chargeAfter commit: publishto the ordinary credit taskcommitCredit task, one transactionquota, usage log,receipt keyed by run idPublish lostbroker refused or worker diedCredit recovery, every 10 minreclaims a stale claim, re-publishesA second deliveryof the same chargereceipt exists: no chargeThe work is done and the user has the answer. Recovery only owes the money.

After the commit, the charge goes to the ordinary credit task. That task writes the quota change, the usage log and a receipt keyed by the child’s run id, in one transaction. A second delivery of the same charge finds the receipt and changes nothing.

If the publish is lost, the charge stays parked on the run row. A credit recovery sweep runs every 10 minutes and publishes it again. The work is done and the user has the answer, so the sweep only has to settle the money.

The review loop found a gap in that sweep. The terminal path claims the charge, commits, and then publishes. In the first version, a crash between the commit and the publish left a charge that was claimed and never billed. The sweep skipped it, because the claim said someone was on it. Now the sweep treats an old claim as abandoned and publishes the charge again. The receipt makes that safe even when the first publish did arrive.

One wake-up, however many children

The parent’s stream ended long ago. A finished child cannot just reply into it.

Child A parks its completion and starts a turn because no turn is streaming; the turn drains A; child B finishes during the turn and stays parked; the turn’s next model step drains B; the turn gives one answer and retires both completion outbox rows when it endsChild AChild BParked setParent turnpark Aidle: start a turndrain {A}, read its findingspark Bstream busy: stay parkednext model step:drain {B}, read its findingsone answer for A and Bturn ends: retire bothcompletion outbox rows
  • The terminal transaction writes a completion outbox row. After the commit, the child adds its id to the conversation’s list of pending completions.
  • Then it checks whether a turn is already streaming. It checks; it does not take the lock. If no turn is streaming, it starts one.
  • A running turn drains the whole list on every model step. So a child that finishes during a turn is folded into it. Five children that finish close together give one answer, not five.
  • The drain removes the ids, but that is not proof that a user saw them. The outbox row is retired only when the turn that showed the completion has ended. A turn that dies before it answers leaves the completion owed, not lost.
  • An outbox sweep every minute re-sends any wake-up whose first dispatch was lost.

The wake-up carries an id, not content. Each drain reads the findings from Postgres. They go to the parent as untrusted data, below the lines the parent may act on, and the parent is told not to follow instructions it finds there. That lowers the risk of a prompt injection through a child. It does not remove it.

Statuses that say who acts next

A background child can outlive its parent’s turn by hours. So a status must say who acts next: a worker, a person, or nobody.

The statuses of a background child: queued, running (the only status that holds a worker), the two waiting statuses that park for a person, cancel_requested, and the terminal statuses succeeded, failed, cancelled and supersededqueuedrunningholds a leaseand a workersucceededfailedwaiting_for_approvalapprove: admitted again,same run idreject: cancelledwaiting_for_resolutionStop is refused. A person sayswhat the tool did: resume(admitted again) or abandoncancel_requestedthe worker still owns itcancelledsupersededclaimapproval neededoutcome unknownStopnext stepworker draining: hand the run backStop while queued or waiting for approval, or reject: cancelled at oncecontinued: a new run, same handleOnly running holds a worker. Both waiting statuses release it.
  • waiting_for_approval: our approval policy stopped a tool call and asked a person. The parent stream is gone, so approval is its own endpoint and not a chat reply. Approve sends the same run id again, and the child passes the caps again. Reject cancels the child.
  • waiting_for_resolution: a tool may have changed something, and nothing can prove whether it did. The child does not guess, and it does not call the tool again, because a second call is the exact failure this state exists to prevent. A person states what the tool really did. Then the child passes the caps again and resumes with that answer, or the person closes it.
  • cancel_requested: the Stop button and the cancel_subagents tool share one policy. A queued child, or one that waits for approval, is cancelled at once. A child that waits for resolution cannot be stopped until a person resolves it. A child that a worker holds gets the request, and the worker writes cancelled at the next step. That way one writer, and only one, makes the child terminal.
  • superseded: a continuation started a new run under the same handle. The old run is kept for the record.

Both waiting statuses release the worker. A child can wait for a person for days and hold no thread.

The agent half

The distributed-systems work was what we expected. The surprise was the agent. An agent that can delegate needs three things. Without each one, the model does something reasonable that does not help the user.

Three needs of a delegating agent, what it does without each, and what our tools tell it insteadThe agent needsWithout itWhat the tool saysA refusal itcan act onretries the same call,or asks the userwhat to do nexta named code, the context,NEXT_ACTIONS(do NOT ask the user)An identity thatsurvives compactionthe handle was in a toolresult the summary dropped:it cannot reach the childlist_subagents returnsevery handle, aliasand status from PostgresA reason toend its turnpolls the status, orholds the turn openwaiting for childrenhandles return now; statewhat started, do not poll;a completion wakes you

A refusal it can act on

A bare error such as “capacity exceeded” gives the model two moves. It retries the same call, or it asks the user what to do next. Every refusal now has a code, the context, and the next action, and it says who decides:

BACKGROUND_SUBAGENTS_DISABLED: new background subagent work is
  unavailable.
Context: requested execution_mode=background; no new background
  child was admitted; existing runs are unchanged.
NEXT_ACTIONS (do NOT ask the user): use foreground only when
  waiting in this turn still satisfies the request; otherwise
  tell the user that background execution is unavailable and
  continue with work this turn can complete.

Some refusals do need the user. A follow-up message to a child that the user stopped is one: only the user can start it again. Those refusals leave out the “do NOT ask the user” prefix. A refusal that says both “do not ask the user” and “the user must decide” makes the model guess.

An identity it can still hold after compaction

A handle that lives only in a tool result does not survive compaction. The summary keeps the story and drops the ids. The parent then knows it started an investigation, but it cannot reach the child that runs it.

So the handle lives in Postgres. list_subagents returns every child’s handle, alias, status and task for the conversation. The model holds the handle, not a run id, so a continuation does not break its reference.

At first, our loop guard, which stopped an agent that repeated the same tool call, blocked list_subagents. Listing again was the correct recovery move, so we exempted it. We have since removed the loop guard.

A reason to end its turn

Give a model a list of running children, and it waits. It polls the status, or it keeps the turn open. The tool description says the opposite, in the same words each time:

execution_mode: background (default) returns handles now,
children run past this turn and each completion wakes you;
state what started, do not poll.

list_subagents repeats it: “Never poll; background completion wakes you.” There is no wait tool. The instruction only works because the outbox makes it true. Ending the turn is safe when a completion can never be lost.

What the review loop found

The build took one week and the review loop took three; three of the bugs it caught were a revoked connection restored on recovery, a crash after commit that stranded a claimed charge, and a child reported finished while its outcome rolled backBuild1 weekReview loop3 weeks0123weeksThree of the bugs the review loop caught:A replay restored a revoked connectionthe recipe was trusted over the agent as it is nowA crash after commit stranded a claimed chargefixed by a sweep that reclaims stale claims, plus a receiptA child was reported finished, but its outcome rolled backit needs a failure at the terminal write to show

The review ran on CloudThinker Code Review. Two of the bugs it found are above: the revoked connection and the stranded charge. The third reported a child as finished while the write that parked its outcome had rolled back. The parent would never have been woken for it. Now the status and the outcome commit in one transaction, or neither does.

None of these bugs shows up when every worker lives. Each one needs a crash, a redelivery or a revoke at one exact moment, so tests that only follow the happy path pass all of them. The fixes came with regression tests that force the moment: a stop that races a claim, a publisher that crashes after the commit, a heartbeat lost at the terminal write.

The open tradeoff

Background children share the worker pool with interactive chat. There is no dedicated queue and no reserved share.

Interactive chat and background children share one worker pool; per-tenant caps bound the children; a dedicated queue is deferred; the revisit trigger is a p95 time to first token above 2 seconds caused by background loadOne shared worker pool, no reserved shareInteractive chat turnsthe user is watching the streamBackground childrenbounded by per-tenant caps:15 per workspace, 30 per organizationDedicated queue and workersdeferred: a fixed split idles threadsRevisit triggerp95 time to first token > 2 s,caused by background load

The admission caps bound background load for each tenant. Nothing reserves a share of the pool for chat, so we watch chat latency to see when the caps stop being enough.

A dedicated queue would split the capacity in a fixed way. Background work would wait while chat threads sat idle, and the other way around. It would also need its own scaling for a feature whose steady volume we do not know yet. So we deferred it, and wrote down when to change our minds: a measured p95 time to first token above 2 seconds in interactive chat, caused by background load.

What we do now

  • A background child is a row in Postgres, not a process. The row owns its status, its lease, its fence and its checkpoint.
  • Only the run id goes on the broker. Every attempt rebuilds the child’s authority from current state, so a replay can narrow it but never widen it.
  • The terminal status, the completion and the charge are written in one transaction. An outbox delivers the completion, and a recovery sweep settles a parked charge. Receipts keyed by run id make a second charge a no-op.
  • The parent answers once per wake-up, however many children finished.
  • Every refusal the model sees names the next action. Every handle the model needs can be listed again after compaction. The model is told to end its turn, and the outbox makes that safe.
  • Background load shares the interactive pool until time to first token says otherwise.

<< Previous: We Tried Every AI Sandbox. Then We Built Our Own.  |  Up: all entries


CloudThinker Engineering · engineering.cloudthinker.io · Last built 2026-11-08 · Plain text version