# Background Subagents in the Cloud: a Row, a Lease and a Fence Instead of a PID

Journey: [Agentic infrastructure](https://engineering.cloudthinker.io/journeys/agentic-infrastructure.md). Entry 2 of 2.
Posted 2026-09-24 by Henry (Duc) Bui. Type: decision.

_Building it took one week. Making it survive crashes, replays and double billing took three._

Background subagents let our main agent, or a specialist agent, hand bounded work to child agents and end its turn at once. The children keep running in the cloud. When they finish, the parent wakes up and answers the user with what they found.

Claude Code and Codex already have this feature on a laptop. But a laptop harness gets a lot for free from the operating system. There is a process to wait on, a local disk to write to, and a signal when a worker dies. Our children run on a shared pool of cloud workers. None of those primitives exist there.

Writing the feature was the small part. Most of the time went into proving three things:

1. A detached run survives a worker that dies mid-step.
2. A replayed message cannot widen what a child is allowed to do.
3. Nobody gets billed twice, and nobody gets billed zero times.

---

## Why the cloud is a different problem

On a laptop, the harness starts a child process and keeps its PID. The PID answers every question: is the child alive, who owns the work, where is its state, did it finish. In the cloud, a Celery worker picks up a message, and any worker can die at any moment. The broker may deliver the same message again. The parent's own turn ended minutes ago.

So every primitive needs a replacement that lives in the database.

![Each laptop primitive and its cloud replacement: a PID becomes a run row, a live process becomes a lease, process ownership becomes a fence token, SIGTERM and restart becomes the reaper, wait() becomes an outbox row, and local memory becomes a Postgres checkpoint](https://engineering.cloudthinker.io/diagrams/agentic-infrastructure/bg-subagents-laptop-vs-cloud.svg)

The distributed-systems half of this was familiar work in an unfamiliar place: leases, fences, outboxes and idempotent delivery. The agent half comes at the end.

## The life of one detached run

Three words in this post have exact meanings:

- A **subagent**, or **child**, is one piece of delegated work. It has a stable handle for its whole life.
- A **run** is one execution of a child, with its own run id and its own row in Postgres.
- A **continuation** is a follow-up task for a finished child. It starts a new run under the same handle, and the old run becomes `superseded`.

A parent calls `spawn_subagents` with `execution_mode: background`. The prompt suggests 2 to 4 children and allows up to 12. The call returns one handle per child, and the parent's turn ends.

![The eight steps of one background child, from the parent call through admission, the broker, the claim, rehydration, the run, the terminal transaction and the wake-up, with the reaper sending the run id through the broker again when a worker dies](https://engineering.cloudthinker.io/diagrams/agentic-infrastructure/bg-subagents-run-lifecycle.svg)

The rest of this post walks through those steps, one failure at a time.

## Admission: all or none

A fan-out asks one question. If we admit half of it, the parent gets half an answer and does not know it. So admission is all or none.

![Admission runs an unlocked preflight, then takes the global lock, recounts both caps, reads credits inside the lock, inserts one run row per child and commits; any failed check refuses all N children](https://engineering.cloudthinker.io/diagrams/agentic-infrastructure/bg-subagents-admission.svg)

- A cheap preflight counts active children without a lock. Most refusals stop here.
- Then one transaction takes a global admission lock, counts again, and reads the credit balance inside the lock. Two fan-outs that arrive together cannot both see room that only one of them fits in.
- A balance we cannot read counts as empty. The gate fails closed.
- Each child gets its own run row, keyed by the id of the spawn call and the child index. If the same call is delivered twice, it finds its own rows and admits nothing new.
- The caps are 15 active children per workspace and 30 per organization. They count `queued` and `running` children only. A child that waits for a person has released its worker, so it does not count. When the person answers, the child passes the same caps again before a worker takes it.

A refusal goes back to the model with a named reason. That matters more than we expected; see [the agent half](#the-agent-half).

## Only the run id crosses the broker

This is the task, trimmed to what matters:

```python
@celery_app.task(
    name="background_subagent.run",
    bind=True,
    acks_late=True,
    time_limit=1800,
    soft_time_limit=1740,
)
async def run_background_subagent(self, run_id: str) -> None: ...
```

Three choices are in those lines.

- **The argument is a run id and nothing else.** The worker loads the model, the tools, the connections and the budget from Postgres. A replayed or forged message can only point at a row that already exists.
- **`acks_late=True` and no retry.** A worker that dies leaves its message unacknowledged, so the broker can deliver it again later. Celery itself never retries the task, because a retry would re-enter a child that may already have done something. The reaper recovers the child instead, and a late copy of the message has to win the same claim as any other worker.
- **The 30-minute limit is far below the broker's one-hour visibility timeout.** With late acknowledgement, a task that outlives the visibility timeout is delivered to a second worker while the first one still runs it. The lease and the fence would keep the second worker out, but the limit stops the case from happening at all.

### Rehydration trusts nothing but the id

The run row holds a versioned recipe for the child. The worker rebuilds the child from it, and checks it against the workspace as it is now.

![The broker message carries only the run id; the worker reads the recipe from the run row and the current workspace state, fails closed on unknown fields, re-authorizes the BYOK key, intersects connections with the agent's current ones, and the child can do what it was granted or less](https://engineering.cloudthinker.io/diagrams/agentic-infrastructure/bg-subagents-replay-authority.svg)

- An unknown recipe version, or an unknown field, fails. Nothing is filled in with a default.
- A bring-your-own-key model is authorized again against the workspace.
- The child's connections are the recipe's connections **intersected with** the ones the spawning agent has now.

The last rule came from the review loop. The first version rebuilt connections from the recipe alone. A user could remove a connection from an agent, and a child that was recovered after that would get the connection back. Nothing about it looked wrong in a single run. It needed a worker to die at the right moment.

## A lease for liveness, a fence for safety

A worker claims a child with one atomic `UPDATE ... RETURNING`. The claim sets a lease that expires in 30 seconds and increments a fence token. While the child runs, a heartbeat renews the lease every 10 seconds. The renewal must match both the owner and the fence.

![Worker A claims with fence 7 and stalls; the lease expires; the reaper re-sends the run id; worker B claims with fence 8 and continues from the checkpoint; worker A wakes, writes its status with fence 7, is rejected, and its failed heartbeat makes it cancel itself](https://engineering.cloudthinker.io/diagrams/agentic-infrastructure/bg-subagents-lease-fence.svg)

The two do different jobs:

- **The lease is for liveness.** It answers "is anybody still working on this?" When it expires, another worker may take the child.
- **The fence is for safety.** A worker that lost its claim is not always dead. It may be stalled, or cut off from the database for a minute. When it comes back, it still believes it owns the child. Its next heartbeat fails because the fence moved, and the worker cancels its own execution. A write that it makes before then to the run row, the tool-call ledger or an approval carries the old fence, and the database rejects it.

A lease without a fence lets two workers write to one child. A fence without a lease never lets anyone take over.

### The reaper

A reaper runs every 2 minutes on a low-priority queue. It finds children with no live lease and re-sends the same run id. The new worker loads the child's LangGraph checkpoint from Postgres, so the child continues from its last step. It does not start again.

- A `queued` child gets 10 minutes of grace first, because a busy pool delays a message without losing it.
- A child gets 3 recovery attempts. After that it fails with `RECOVERY_EXHAUSTED`, and the parent hears about it. A child that retries forever looks like live work that never moves.
- The reaper reserves an attempt with an atomic `UPDATE` before it sends anything. Two reapers that scan the same child can each spend an attempt, but the count never passes 3.
- A worker that is shutting down does not start a child. It hands the run back, and another worker takes it.

Before it claims, a worker also skips a child that is already terminal, has a stop requested, or waits for a person to resolve a tool call. A child that waits for approval is different: Approve sends its run id again, and a worker claims it then.

## Nobody gets billed twice

A child's charge is written in the same transaction that makes the child terminal. That transaction writes three things: the status, the completion outbox row and the parked charge.

![One terminal transaction writes the status, the completion outbox row and the parked charge; after commit the charge is published to the credit task; a lost publish is recovered by credit recovery every 10 minutes, which reclaims the stale claim and re-publishes; a second delivery finds the receipt and charges nothing](https://engineering.cloudthinker.io/diagrams/agentic-infrastructure/bg-subagents-billing-once.svg)

After the commit, the charge goes to the ordinary credit task. That task writes the quota change, the usage log and a receipt keyed by the child's run id, in one transaction. A second delivery of the same charge finds the receipt and changes nothing.

If the publish is lost, the charge stays parked on the run row. A credit recovery sweep runs every 10 minutes and publishes it again. The work is done and the user has the answer, so the sweep only has to settle the money.

The review loop found a gap in that sweep. The terminal path claims the charge, commits, and then publishes. In the first version, a crash between the commit and the publish left a charge that was claimed and never billed. The sweep skipped it, because the claim said someone was on it. Now the sweep treats an old claim as abandoned and publishes the charge again. The receipt makes that safe even when the first publish did arrive.

## One wake-up, however many children

The parent's stream ended long ago. A finished child cannot just reply into it.

![Child A parks its completion and starts a turn because no turn is streaming; the turn drains A; child B finishes during the turn and stays parked; the turn's next model step drains B; the turn gives one answer and retires both completion outbox rows when it ends](https://engineering.cloudthinker.io/diagrams/agentic-infrastructure/bg-subagents-wake-up.svg)

- The terminal transaction writes a completion outbox row. After the commit, the child adds its id to the conversation's list of pending completions.
- Then it checks whether a turn is already streaming. It checks; it does not take the lock. If no turn is streaming, it starts one.
- A running turn drains the whole list on every model step. So a child that finishes during a turn is folded into it. Five children that finish close together give one answer, not five.
- The drain removes the ids, but that is not proof that a user saw them. The outbox row is retired only when the turn that showed the completion has ended. A turn that dies before it answers leaves the completion owed, not lost.
- An outbox sweep every minute re-sends any wake-up whose first dispatch was lost.

The wake-up carries an id, not content. Each drain reads the findings from Postgres. They go to the parent as untrusted data, below the lines the parent may act on, and the parent is told not to follow instructions it finds there. That lowers the risk of a prompt injection through a child. It does not remove it.

## Statuses that say who acts next

A background child can outlive its parent's turn by hours. So a status must say who acts next: a worker, a person, or nobody.

![The statuses of a background child: queued, running (the only status that holds a worker), the two waiting statuses that park for a person, cancel_requested, and the terminal statuses succeeded, failed, cancelled and superseded](https://engineering.cloudthinker.io/diagrams/agentic-infrastructure/bg-subagents-status-machine.svg)

- **`waiting_for_approval`**: our approval policy stopped a tool call and asked a person. The parent stream is gone, so approval is its own endpoint and not a chat reply. Approve sends the same run id again, and the child passes the caps again. Reject cancels the child.
- **`waiting_for_resolution`**: a tool may have changed something, and nothing can prove whether it did. The child does not guess, and it does not call the tool again, because a second call is the exact failure this state exists to prevent. A person states what the tool really did. Then the child passes the caps again and resumes with that answer, or the person closes it.
- **`cancel_requested`**: the Stop button and the `cancel_subagents` tool share one policy. A `queued` child, or one that waits for approval, is cancelled at once. A child that waits for resolution cannot be stopped until a person resolves it. A child that a worker holds gets the request, and the worker writes `cancelled` at the next step. That way one writer, and only one, makes the child terminal.
- **`superseded`**: a continuation started a new run under the same handle. The old run is kept for the record.

Both waiting statuses release the worker. A child can wait for a person for days and hold no thread.

## The agent half

The distributed-systems work was what we expected. The surprise was the agent. An agent that can delegate needs three things. Without each one, the model does something reasonable that does not help the user.

![Three needs of a delegating agent, what it does without each, and what our tools tell it instead](https://engineering.cloudthinker.io/diagrams/agentic-infrastructure/bg-subagents-agent-contract.svg)

### A refusal it can act on

A bare error such as "capacity exceeded" gives the model two moves. It retries the same call, or it asks the user what to do next. Every refusal now has a code, the context, and the next action, and it says who decides:

```text
BACKGROUND_SUBAGENTS_DISABLED: new background subagent work is
  unavailable.
Context: requested execution_mode=background; no new background
  child was admitted; existing runs are unchanged.
NEXT_ACTIONS (do NOT ask the user): use foreground only when
  waiting in this turn still satisfies the request; otherwise
  tell the user that background execution is unavailable and
  continue with work this turn can complete.
```

Some refusals do need the user. A follow-up message to a child that the user stopped is one: only the user can start it again. Those refusals leave out the "do NOT ask the user" prefix. A refusal that says both "do not ask the user" and "the user must decide" makes the model guess.

### An identity it can still hold after compaction

A handle that lives only in a tool result does not survive compaction. The summary keeps the story and drops the ids. The parent then knows it started an investigation, but it cannot reach the child that runs it.

So the handle lives in Postgres. `list_subagents` returns every child's handle, alias, status and task for the conversation. The model holds the handle, not a run id, so a continuation does not break its reference.

At first, our loop guard, which stopped an agent that repeated the same tool call, blocked `list_subagents`. Listing again was the correct recovery move, so we exempted it. We have since removed the loop guard.

### A reason to end its turn

Give a model a list of running children, and it waits. It polls the status, or it keeps the turn open. The tool description says the opposite, in the same words each time:

```text
execution_mode: background (default) returns handles now,
children run past this turn and each completion wakes you;
state what started, do not poll.
```

`list_subagents` repeats it: "Never poll; background completion wakes you." There is no wait tool. The instruction only works because the outbox makes it true. Ending the turn is safe when a completion can never be lost.

## What the review loop found

![The build took one week and the review loop took three; three of the bugs it caught were a revoked connection restored on recovery, a crash after commit that stranded a claimed charge, and a child reported finished while its outcome rolled back](https://engineering.cloudthinker.io/diagrams/agentic-infrastructure/bg-subagents-build-vs-review.svg)

The review ran on CloudThinker Code Review. Two of the bugs it found are above: the revoked connection and the stranded charge. The third reported a child as finished while the write that parked its outcome had rolled back. The parent would never have been woken for it. Now the status and the outcome commit in one transaction, or neither does.

None of these bugs shows up when every worker lives. Each one needs a crash, a redelivery or a revoke at one exact moment, so tests that only follow the happy path pass all of them. The fixes came with regression tests that force the moment: a stop that races a claim, a publisher that crashes after the commit, a heartbeat lost at the terminal write.

## The open tradeoff

Background children share the worker pool with interactive chat. There is no dedicated queue and no reserved share.

![Interactive chat and background children share one worker pool; per-tenant caps bound the children; a dedicated queue is deferred; the revisit trigger is a p95 time to first token above 2 seconds caused by background load](https://engineering.cloudthinker.io/diagrams/agentic-infrastructure/bg-subagents-shared-pool.svg)

The admission caps bound background load for each tenant. Nothing reserves a share of the pool for chat, so we watch chat latency to see when the caps stop being enough.

A dedicated queue would split the capacity in a fixed way. Background work would wait while chat threads sat idle, and the other way around. It would also need its own scaling for a feature whose steady volume we do not know yet. So we deferred it, and wrote down when to change our minds: a measured p95 time to first token above 2 seconds in interactive chat, caused by background load.

## What we do now

- A background child is a row in Postgres, not a process. The row owns its status, its lease, its fence and its checkpoint.
- Only the run id goes on the broker. Every attempt rebuilds the child's authority from current state, so a replay can narrow it but never widen it.
- The terminal status, the completion and the charge are written in one transaction. An outbox delivers the completion, and a recovery sweep settles a parked charge. Receipts keyed by run id make a second charge a no-op.
- The parent answers once per wake-up, however many children finished.
- Every refusal the model sees names the next action. Every handle the model needs can be listed again after compaction. The model is told to end its turn, and the outbox makes that safe.
- Background load shares the interactive pool until time to first token says otherwise.

---

Previous: [We Tried Every AI Sandbox. Then We Built Our Own.](https://engineering.cloudthinker.io/journeys/agentic-infrastructure/01.md)
Web version: https://engineering.cloudthinker.io/journeys/agentic-infrastructure/02/
