CloudThinker Engineering

Agentic system design, entry 2: Eager Tool Calling: How We Cut Agent Latency by 50% on Long Tool Chains

Henry Bui — Wed, 15 Apr 2026 00:00:00 GMT

1. Introduction — The Latency Nobody Was Measuring

An eight-tool agent task in our staging environment clocked in at twenty-four seconds. The model was fast. The individual tools were fast. The wall clock was slow. Where did the time go?

We dropped the trace into a flame graph and the answer was embarrassing. For the first four seconds the model streamed tokens while every tool sat idle. Then the model finished, fired all eight tools in parallel, and spent another twenty seconds waiting for them. Two phases, stacked end to end, with no overlap between them.

We rewrote that codepath. The same request now finishes in roughly five seconds — the four-second stream and the tool execution happen in the same four seconds instead of consecutively. Across a representative sample of production traces, median end-to-end task completion is 50% faster, and longer tool chains see proportionally larger wins.

The technique is called eager tool calling (internally we call it tool-call pipelining). Think of it as CPU instruction pipelining for agents: just as a modern CPU doesn’t wait for one instruction to retire before decoding the next, an eager runtime doesn’t wait for the model to finish before starting the tools it has already described. The idea is small. The engineering was not. Here is what it is, how it works, the production bugs that taught us what not to do, and the numbers we measured.

2. The Problem: Where Agent Latency Actually Hides

The common intuition is that parallel tool calling already solved this. It didn’t. Parallel tool calling is necessary but not sufficient.

The classic agent loop

A single agent turn looks like this:

Model reasons over the context and begins streaming a response.
The response contains one or more tool_use blocks.
Runtime waits for message_stop.
Runtime executes the tools.
Runtime collects the tool_result blocks and starts the next turn.

Total wall clock for one turn = stream time + tool time, added serially. The tool layer is a passive consumer that refuses to start work until the producer has completely finished.

The parallel tool calling half-fix

Modern APIs — Anthropic, OpenAI, Bedrock — let the model emit multiple tool_use blocks in a single assistant message, and mature runtimes run those blocks concurrently.

That change moves the tool phase from sum of tool durations to max of tool durations. Genuinely valuable. On an eight-tool turn with tools averaging 2.5 seconds, parallel tool calling takes ~2.5 seconds for the tool phase instead of 20. Big win.

But the stream phase still happens first. The tools still wait for message_stop. If the model takes four seconds to emit its response, those four seconds are pure dead time on the tool side.

What we wanted

Tools that start running the instant the model finishes emitting them, while the model continues streaming the rest of the message. Not tools-parallel-with-tools. Tools-parallel-with-generation.

Parallel tool calling overlaps tools with each other. Eager tool calling overlaps tools with the model itself.

3. Three Eras of Tool Calling

Era	Concurrency	When tools start	Wall clock
Sequential	None	After each prior tool finishes	Σ(stream + all tools)
Parallel	Tools with tools	After `message_stop`	stream + max(tool)
Eager (pipelined)	Tools with tools and tools with generation	The moment each tool block finishes streaming	max(stream, max(tool))

The move from sequential to parallel collapses one dimension of latency. The move from parallel to eager collapses the other. Time goes from sum to max in both directions.

4. How Eager Tool Calling Works

The key insight — the seal

Each tool_use block the model emits has a stable tool_call_id. During streaming, the runtime sees chunks arrive one at a time. Most chunks carry a tool_call_id; partial argument chunks carry only an index.

Here is the observation that makes eager tool calling possible: the moment a chunk arrives with a tool_call_id different from the previous one, the previous tool call is definitionally complete. The model has moved on. Its arguments are fully accumulated. It is safe to execute right now, even though the assistant message is still streaming.

We call that transition the seal event. A tool is sealed the instant the model starts streaming a different tool — or when the message itself ends.

How it works in practice

Each chunk that arrives with a new tool ID is proof the previous tool is fully assembled — we fire it immediately, while the model keeps streaming the rest of the message.

Visualizing the overlap

In a correct eager execution, the tool lanes are not end-to-end. They overlap:

stream : [==================================]
tool A :   [=========]            ← fires when B's id arrives
tool B :       [=========]        ← fires when C's id arrives; runs concurrent with A
tool C :           [=========]    ← fires at message_stop; runs concurrent with A + B

Drop a vertical line at the 50% mark and it cuts through all three bars. That is the signature of eager tool calling. If your “eager” diagram shows tools end-to-end, you have implemented sequential tool calling under a different name.

5. Inside CloudThinker’s Stream Handler

The dispatch logic is straightforward to describe, even if the implementation required care.

We track the most-recent tool call ID seen in the stream. The moment a new ID arrives, the previous tool is complete — we dispatch it immediately to a background worker pool without waiting for the rest of the message. That’s the entire seal trigger: one state comparison per chunk.

Partial argument chunks don’t carry an ID — they only carry a positional index. So we maintain a small index-to-ID map that lets us route each partial chunk back to the right accumulation buffer, keeping the arguments intact even when multiple tools are streaming interleaved.

The dispatch is fire-and-forget by design. If a background worker raises an exception, it’s caught and surfaced as an error tool_result on the next turn. The stream reader never sees it, and the other tools keep running.

6. Production Lessons — What We Learned the Hard Way

The first surprise was tool retraction. The model occasionally emits a tool call mid-stream, reconsiders, and replaces it. Rare, but real. A tool that has already fired eagerly can’t be un-fired. We added a per-tool idempotent flag: non-idempotent operations — payments, destructive commands, outbound emails — fall back to the classic path and only fire after message_stop. The eager fast path is reserved for reads and safe operations.

The second was cancellation. When the user interrupts or the model emits a stop sequence mid-stream, every in-flight eager tool needs to cancel cleanly. Leaking a tool execution into a dead conversation is both a correctness bug and a resource leak. We tied each dispatch to a cancellation scope whose lifetime matches the stream reader — stream dies, scope cancels, tools abort.

Then a flaky S3 list operation crashed an entire agent turn for a minute before we realized we had no exception boundary around individual tool dispatches. One failing tool was bringing down the stream reader. Now every sealed dispatch runs in its own exception boundary; failure produces an error tool_result and nothing else propagates.

Finally, eager execution hides behavior inside the stream in a way classic execution doesn’t. “Why is tool X slow?” becomes unanswerable without knowing when it was sealed versus when it finished. Every seal event now emits an observability span with seal_latency_ms, tool_call_id, and conversation_id. When something misbehaves, the timeline tells you exactly which stage was slow.

7. The Numbers

We benchmarked 16 workloads representative of real agent traffic, comparing sequential baseline, parallel tool calling, and eager tool calling. P50 wall clock from request-received to final assistant message. The numbers below come from the open-source benchmark in eager-tools — synthetic, deterministic, reproducible by anyone via make bench.

#	Workload	Tools	Sequential p50	Parallel p50	Eager p50	Speedup vs parallel	Speedup vs sequential
1	Analytics query	3	4.9s	3.5s	2.9s	1.21×	1.69×
2	Search & retrieval	4	5.7s	4.2s	3.5s	1.20×	1.63×
3	Customer support	6	8.6s	5.5s	4.5s	1.22×	1.91×
4	Deploy preflight	7	29.9s	17.0s	14.2s	1.20×	2.11×
5	Incident triage	9	17.6s	9.5s	6.5s	1.46×	2.71×
6	Security sweep	10	37.8s	14.0s	10.2s	1.37×	3.71×
7	Research synthesis	10	22.5s	10.0s	8.0s	1.25×	2.81×
8	Lead enrichment	7	13.4s	7.0s	5.5s	1.27×	2.44×
9	Content moderation	8	19.5s	13.0s	10.0s	1.30×	1.95×
10	DB migration	9	20.9s	11.0s	8.9s	1.24×	2.35×
11	Release notes	10	19.6s	9.5s	6.7s	1.42×	2.93×
12	Legal review	13	28.4s	11.5s	9.4s	1.22×	3.02×
13	Sales outreach	14	23.6s	11.0s	8.6s	1.28×	2.75×
14	Ad campaign sweep	15	30.4s	11.5s	8.8s	1.31×	3.46×
15	Invoice processing	6	13.5s	7.5s	5.2s	1.44×	2.60×
16	Onboarding flow	7	13.1s	7.5s	5.0s	1.50×	2.62×

The synthetic harness removes network jitter so the comparison isolates the dispatch strategy — speedup vs parallel ranges from 1.20× to 1.50×, median ~1.28×. In production, where tail latency and provider variance compound, end-to-end task completion is ~50% faster on the median trace, and longer tool chains pull further ahead. Treat the OSS bench as the lower-bound version anyone can reproduce on a laptop; production wins are larger.

Cost impact

Output tokens per task: effectively unchanged (same reasoning, same answer)
Net cost reduction: ~35% on tool-heavy workloads — fewer retries from timeouts, more conversations completing inside the cache TTL window, and faster end-to-end task completion freeing up agent slots.

Where the speedup comes from

Two stacked effects:

Generation / execution overlap. The biggest chunk. Every millisecond of stream time that used to sit idle on the tool side now has a tool running in parallel.
Fewer turns. A task that used to take three model turns often completes in one. Each saved turn saves prefill, network round-trip, and reasoning overhead.

8. When NOT to Use It

Being honest about the limits.

Fast tools (sub-50ms). If your tools finish in milliseconds, there is nothing to hide behind the stream. Seal/dispatch overhead exceeds the latency saved. Don’t bother.
Sequentially dependent tools. If tool B needs tool A’s result to even be formulated, the model won’t emit B until A returns. No pipeline opportunity; eager and classic are identical.
Non-streaming backends. You can’t seal per block without incremental parsing. If your provider or gateway buffers the full response before forwarding, eager tool calling is impossible until that changes.
Non-idempotent tools. Already covered above. Destructive operations, payments, outbound messages — these stay on the classic path.

9. A Mental Model That Helps

Eager tool calling is CPU instruction pipelining for agents. The agent runtime is the CPU, the model stream is the instruction fetch, the tools are the execution units, and the seal event is the register-ready signal that lets execution begin before the rest of the batch is decoded. Once that analogy clicks, the rest of the design falls out of it naturally.

10. Try It Yourself

Build it yourself

Building this requires a streaming SSE parser, a seal detector, an async executor pool tied to the stream’s lifetime, per-tool idempotency flags, and observability spans on every seal event. The open-source eager-tools library ships all of this.

Closing

Eager tool calling is not a novel idea in the abstract. CPUs have been pipelining instructions since the 1980s. What’s novel is that the agent ecosystem spent two years treating streaming as a UX-only feature — something you do to make tokens appear live in a chat window — rather than as an opportunity to parallelize execution against generation.

For any production agent running multiple tools per turn, this pattern is not optional. It is the difference between an impressive demo and a system fast enough to replace a human operator.

First published on cloudthinker.io on April 15, 2026.

Generative UI, entry 1: Generative UI in Production: Lessons from a Pydantic-to-DSL Migration

Henry Bui — Mon, 06 Apr 2026 00:00:00 GMT

Our agent dashboards used to take 30-40 seconds and cost roughly $0.50 per report. Today they stream in under 10 seconds for around $0.08 — and they’re built on the same design system as the rest of the product. This is the story of a migration we expected to take a week and ended up rethinking from first principles.

A year ago, asking a CloudThinker agent for a cost dashboard meant watching a spinner. The agent would call a dashboard() tool, the model would fill in a strict Pydantic schema field by field, and 30-something seconds later — once the entire JSON tree validated — a fully-formed dashboard would snap into place. It looked great. It used the platform’s design system. It was also slow enough that most users would tab away and forget what they’d asked for.

Today, the dashboard streams in. KPI cards render first with current-month totals. A line chart fills in as the agent emits it. A service breakdown appears as a bar chart. Everything still matches CloudThinker’s design system. Total time: under 10 seconds. Total cost: about $0.08. The chart is interactive because it is the product, not a snapshot of it — click through to the expensive region and you drill in immediately.

Metric	Was	Now	Change
Wall-clock latency	30–40s	~10s	~75% faster end-to-end
Cost per report	~$0.50	~$0.08	~84% cheaper per artifact
Render mode	Atomic PNG	Streaming	Progressive, line by line
Design consistency	Drifted	Locked	Enforced by construction

This post is about how we got from the first version to the second — and, more importantly, why the change wasn’t a micro-optimization. It was a category shift the whole industry made at the same time, and it took us two architectures to land on the right one.

Act 0 — The Two Detours We Skipped

Before talking about what we built, it’s worth being honest about what we didn’t build. When we first sat down with the “agent generates dashboards” problem, the two obvious paths were already on the whiteboard. We crossed both of them out fast.

Matplotlib PNGs. The most popular pattern in 2023. Let the LLM write Python, run it in a sandbox, capture the PNG. Research from the MatPlotAgent paper (arXiv 2402.11453) is worth quoting: it “struggles to handle complex charts that require fine-grained numerical accuracy and precise visual-textual alignment of chart labels with plot components.” A separate study, Are LLMs Ready for Visualization? (arXiv 2403.06158), documented that “numerous attempts to modify the default configuration of scatterplots and bubble charts were unsuccessful.” Five runs of the same prompt give five different results. There’s a 4-8 second cold-start tax on import matplotlib. PNGs aren’t interactive, can’t be themed without a fight, and are effectively invisible to screen readers — the ACM DIS 2025 GenUI Study called accessibility “a frequently-mentioned constraint” for generated output. And every chart means executing LLM-generated Python in a sandbox. The best way to not have a sandbox escape is to not execute the code in the first place.

The agent writes Python, a sandbox executes matplotlib, and a PNG embeds in chat. Five specific failure modes compound: design drift, stochastic output, cold-start overhead, zero interactivity, and accessibility gaps.

Raw HTML / JSX freestyle. “Modern frontier models are great at React and Tailwind. Why don’t we just let the agent write the UI directly?” Everyone has this idea. Hardik Pandya’s essay Expose your design system to LLMs captures why it’s a trap: LLMs “fabricate token names, drift on values within a session, lose all context between sessions, and never notice when the upstream library ships breaking changes.” Token economics collapse — a single dashboard becomes 2,000-4,000 tokens of markup before a single data point. Half-streamed HTML is broken HTML, so progressive rendering is impossible. Every dashboard becomes a one-off; the design system is a suggestion, not a contract.

And then there’s the security bomb.

OWASP LLM05 — Improper Output Handling

Our injection vector wasn’t hypothetical. Cloud resource names are user-controlled. A customer can tag an EC2 instance with an onerror attribute payload, and the next time our agent generates a dashboard referencing that resource, the tag flows through the LLM’s output straight into dangerouslySetInnerHTML.

The agent isn’t even “attacked” — it’s a pass-through for attacker-controlled strings in the most structurally dangerous way possible. The OWASP guidance is blunt: treat LLM output as untrusted user input. The only durable fix is to make it structurally impossible for the schema to represent a <script> tag in the first place.

OWASP LLM05: Improper Output Handling is officially catalogued in the OWASP Top 10 for LLM Applications. Auth0’s writeup puts it bluntly: “Improper Output Handling is the New XSS.” The PortSwigger Web Security Academy LLM lab walks through the exact attack: a model is coaxed — directly or via indirect prompt injection — into emitting an onerror payload that ends up in dangerouslySetInnerHTML. The OWASP guidance is blunt: treat LLM output as untrusted user input. The only way to be sure a model can’t emit a dangerous tag is to make the schema incapable of representing one.

We skipped both detours. What we picked instead seemed clever at the time — and it was, until it wasn’t.

Act 1 — The Pydantic Era

A year ago, we shipped a dashboard() tool. The agent calls it with strictly typed Pydantic widgets — KPI cards, charts, tables — placed on a 12-column grid. The frontend renders them with the same React components the rest of the product uses. No Python. No HTML. No PNG. Just a typed object emitted through OpenAI’s structured-output mode (or Bedrock’s equivalent), with the model’s decoding constrained against the schema.

The shape, lightly trimmed:

class KpiCardWidget(KPICard):
    layout: LayoutConfig          # x, y, w, h on a 12-col grid
    type: Literal["kpi_card"]
    section_id: str | None

class ChartWidget(ChartStructuredOutput):
    layout: LayoutConfig
    type: Literal["chart"]
    # categories, datasets, axes, target_line, threshold_zones...

class TableWidget(TableStructuredOutput):
    layout: LayoutConfig
    type: Literal["table"]
    # columns, rows, summary_row...

DashboardWidget = Annotated[
    KpiCardWidget | TableWidget | ChartWidget,
    Discriminator("type"),
]

class Dashboard(BaseModel):
    title: str
    widgets: list[DashboardWidget]
    sections: list[DashboardSection] | None
    time_context: TimeContext | None

The agent emits a Dashboard object via tool call. Discriminated unions guarantee every widget has a known shape. Validators enforce layout rules — KPI cards must be in row 0, chart width must be at least 3 columns, widgets sharing a row must belong to the same section — and feed structured error messages back to the model when something’s off. The renderer is a single <DashboardTab> component that walks the tree and maps each typed widget to a real platform component (Highcharts inside Card containers, native tables, themed KPI cards).

This was a smart starting point. We knew matplotlib and HTML freestyle were dead ends, and constrained decoding gave us a real escape hatch. Specifically:

Design consistency for free. Every chart is a Highcharts <LineChart> from the design system. There is no “agent invented a new shade of primary blue” because the agent is selecting components, not styling them.
Zero XSS surface. The agent physically cannot emit <script>. The schema has no field that lets it.
No code execution. No sandbox, no import matplotlib, no Python interpreter on the hot path.
Validation is structural. Bad widget = clean error message back to the agent (“KPI card width must be 3, 4, or 6”). The agent fixes itself most of the time without a retry loop.
Auditable. Dashboards persist as typed JSON in PostgreSQL. Every report a customer ever saw is reconstructable.

For about ten months, this was our answer. It shipped, customers used it, and it didn’t break. But by month six the cracks were obvious to anyone watching the latency dashboard.

Token economics. A four-widget dashboard was 1,500-3,000 tokens of JSON before a single data point. Discriminated unions are verbose by nature: every widget repeats type, layout: {x, y, w, h}, section_id, and the full nested chart config. Multiply by a long-running investigation that emits multiple reports, multiply by users, multiply by frontier-model output pricing. The line item showed up.

No real streaming. This is the one that hurt the most. Constrained decoding makes the LLM wait until the entire JSON tree parses cleanly before it commits a single token to the output. Structured-output mode (Pydantic schemas, JSON Schema, function-call parameters — they’re all the same shape) fundamentally trades streamability for validity. The user sees a spinner for 30+ seconds, then the whole dashboard appears at once. Wall-clock latency was OK by 2025 standards. Perceived latency was the worst part of the product.

Schema rigidity. Want a new chart type? Add a discriminated variant. Update the validator. Regenerate frontend types via pnpm gen:local. Ship a backend release. Ship a frontend release. A new component took a sprint. A new layout pattern took half a sprint. Every time the design team wanted to try something, the answer was “yes, in the next release.”

One artifact per conversation. The tool was modeled around create_dashboard / add_widgets / update_widgets. A single artifact lifecycle. Agents that wanted to emit multiple small reports in a single thread had to overwrite the previous one or skip.

Pixel arithmetic instead of intent. A Stack(KPI, KPI, Chart) layout had to be expressed as four absolute coordinates ({x: 0, y: 0, w: 4}, {x: 4, y: 0, w: 4}, {x: 8, y: 0, w: 4}, {x: 0, y: 1, w: 12, h: 3}). The agent burned tokens reasoning about grid math instead of describing what it wanted to show. We had a half-page of layout examples in the tool description that the agent had to re-read on every call.

End-of-Act-1 scoreboard: 30-40 seconds wall-clock, roughly $0.50 per report. Functionally correct, design-system-consistent, secure — and unit economics that didn’t pencil out for what we wanted to build next.

Act 2 — The DSL We’d Been Dreaming About

We’d been talking about “what if the agent emitted a tiny language instead of a typed JSON tree” for months. We even sketched a grammar on a whiteboard once. Then we kept shipping other things, and the dashboard kept being good enough.

What changed in early 2026: the rest of the industry caught up to where we’d been doodling. Generative UI became its own category, with multiple serious implementations converging on the same basic idea — and several of them shipped to production while we were heads-down on other features.

The Generative UI landscape, 2024–2026

#	Project	What it is
01	OpenUI Lang (our choice)	Compact, line-oriented DSL. Published benchmarks show 67.1% fewer tokens than Vercel JSON-Render.
02	Vercel AI SDK	streamUI with React Server Components. As of late 2025, RSC development is officially paused; Vercel recommends AI SDK UI for production.
03	LangChain / LangGraph	Tool calls mapped to pre-registered React components with type-safe streaming via assistant-ui and useStream().
04	Thesys C1	OpenAI-compatible endpoint that outputs UI components instead of text. Drop-in for chat interfaces that want to become dashboards.
05	CopilotKit AG-UI	Event/state protocol between agents and UI. 120K+ weekly installs; adopted by Google, AWS, LangChain, Microsoft, Mastra, PydanticAI.
06	Google A2UI	Declarative spec from Google Research for agent-generated UIs. Part of the same wave as AG-UI and OpenUI.

Vercel AI SDK 3.0 introduced streamUI in early 2024 — stream React Server Components from LLM tool calls. Worth weighing: per Vercel’s own discussion thread, “Development of AI SDK RSC is paused. We recommend using AI SDK UI.” Even Vercel couldn’t make “stream React components from the server” work well enough to recommend for production.
LangChain / LangGraph Generative UI maps tool calls to pre-registered React components with type-safe streaming. The assistant-ui library provides an out-of-the-box renderer with progressive updates during long-running graph execution.
Thesys C1 is an OpenAI-compatible endpoint that outputs UI components instead of text. Ask it “show me monthly revenue trends” and get a rendered line chart instead of a paragraph. See also their guide to building Generative UI applications.
CopilotKit AG-UI is a bidirectional agent↔UI event protocol — adopted by Google, LangChain, AWS, Microsoft, Mastra, and PydanticAI, with over 120K weekly installs as of early 2026.
Google A2UI is Google’s declarative spec for agent-generated UIs. Google Research has also published their own generative UI work.
OpenUI Lang takes the most opinionated line: a compact, line-oriented DSL designed for LLMs to stream UI structure directly. Their published benchmarks report 60–67% fewer tokens than JSON-based alternatives and up to 3× faster render on deeply nested UIs.

CopilotKit’s framing is useful. They describe three patterns of Generative UI: Controlled (high structure, low freedom), Declarative (shared control), and Open-ended (agents produce arbitrary UI). Our Pydantic Era was firmly Controlled — we just picked a payload format that didn’t stream. The fix was to keep the Controlled posture and swap the payload.

The pattern underneath all of them

Peel back the branding and these projects agree on more than they disagree on. Every production-leaning approach has landed on the same four properties:

A closed component vocabulary. The agent can only emit things the renderer already knows how to draw. No arbitrary markup.
A typed, structured payload. JSX, JSON, YAML, or a custom DSL — but always something a parser can validate before it reaches the DOM.
Streaming-friendly grammar. Partial output has to be meaningful. Half an HTML tag is broken; half a tool call or half a DSL block should still render something.
Renderer-owned styling. The agent describes intent (status: "success"); the renderer decides which shade of green.

The shape of the payload varies — tool calls (Vercel/LangChain), JSON schemas (Thesys), event protocols (AG-UI/A2UI), or line-oriented DSLs (OpenUI). But the contract is identical: agents describe, renderers render. The whole category moved this direction because the alternative — letting the agent write the view layer — was strictly worse on every axis that mattered.

OpenUI Lang’s published benchmark shows 67.1% fewer tokens than Vercel JSON-Render, 65.4% fewer than Thesys C1 JSON, and 61.4% fewer than YAML across seven real-world UI scenarios. For deeply nested dashboards the gap grows to 3.0× — the main reason we picked it for CloudThinker’s Artifact system. Source: openui.com/docs/openui-lang/benchmarks.

The token economics drive a lot of the architectural choices. JSON is verbose. YAML is slightly less so. Line-oriented DSLs are the most compact of the common options, and that gap compounds: shorter payloads mean smaller system prompts, fewer output tokens, faster streaming, and more predictable costs. A dashboard that a JSON-based renderer takes ~14 seconds to finish can stream in ~5 seconds with a DSL grammar designed for progressive parsing.

Why we didn’t roll our own

Designing a DSL is the easy part. Building the parser, the validator, the streaming render path, the error messages, the few-shot examples, the documentation — that’s months. Every serious GenUI project has converged on the same four properties independently. Picking off the shelf was strictly better than re-deriving the same wheel.

We evaluated against three things that mattered to us, in order:

Token economics. We already knew the Pydantic JSON was too verbose. Whatever we picked had to be measurably smaller.
Streaming-first grammar. Partial parse must render something — that was the lesson from the Pydantic era’s spinner problem. If the format requires a closing brace before it’s valid, it disqualifies itself.
Maps cleanly to a closed component vocabulary we already own. We didn’t want to throw away the design-system components from the Pydantic era. Whatever we picked had to be a payload format we could point at our existing renderers.

OpenUI Lang scored highest on all three. The 60-67% token reduction over JSON-based payloads was the visible win, but the hidden win was the line-oriented grammar — each kpi = KPICard(...) is an independently parseable statement. Half a dashboard renders as half a dashboard, not a spinner. We adapted the OpenUI vocabulary to our FinOps-flavored component palette and built our Artifact system on top.

Act 3 — How CloudThinker’s Artifact System Works Now

The Artifact system is the third iteration, not the answer that fell from the sky. The agent calls an artifact() tool. The tool accepts structured parameters — type, title, description, and a body containing a compact, line-oriented DSL block (inspired by OpenUI Lang, adapted to a FinOps-flavored component palette). The body is validated against a component whitelist, stored in PostgreSQL, and streamed to the frontend.

The agent emits typed DSL inside an artifact() tool call. The parser enforces a component whitelist. The body is stored as plain text in PostgreSQL and rendered to multiple surfaces — React for web, WeasyPrint for PDF — from the same source.

A concrete example. This is the body an agent emits for a Q4 cost dashboard — note how each line is a complete component assignment, parseable and renderable on arrival:

kpi_spend = KPICard("Total Spend", "$12,450", "-8%", "success")
kpi_save  = KPICard("Savings",     "$2,400",  "+12%", "success")

trend = LineChart(
  ["Jan", "Feb", "Mar"],
  [Series("Cost", [13500, 12450, 11800])]
)

breakdown = BarChart(
  ["EC2", "RDS", "S3"],
  [Series("Cost", [5200, 3100, 1800])]
)

kpi_row = Stack([kpi_spend, kpi_save], "row")
root = Stack([
  kpi_row,
  Card([CardHeader("Cost Trend", "Monthly spend trajectory"), trend]),
  Card([CardHeader("By Service", "Current month breakdown"), breakdown])
])

That’s the entire payload. Compare it to the equivalent Pydantic Era JSON — same dashboard, but with type discriminators, layout: {x, y, w, h} per widget, full chart-axis configs, and section_id references. Same information, several times the tokens.

The component palette is deliberately small. Agents can emit:

Layout: Stack, Card, CardHeader, Divider
Data: KPICard, Table, TextContent, Callout, Tag
Charts: BarChart, LineChart, AreaChart, PieChart, RadarChart, ScatterChart, SingleStackedBarChart
Advanced: GaugeChart, Treemap, Heatmap, FeatureMatrix, ProsCons, ScoreTable

That’s the entire vocabulary. An agent physically cannot emit a <script> tag, a rogue <div>, a fabricated color token, or a chart library we haven’t vetted. The OWASP LLM05 attack surface is gone by construction, just like in the Pydantic era — but now with progressive streaming and ~70% fewer tokens.

The frontend renders the DSL with a registered component library (Highcharts for charts, native components for everything else). The same DSL also feeds a WeasyPrint-backed HTML renderer that produces PDFs — same body, same output, different surface. Write once, render everywhere.

Dimension	Matplotlib (detour)	Raw HTML (detour)	Pydantic Era	Artifact Era
Output format	Binary PNG	HTML markup	Typed JSON tree	Line-oriented DSL
Streaming	No — atomic	Half-HTML is broken	Wait for valid JSON	Yes — line by line
Interactivity	None	Arbitrary (risky)	Componentized	Componentized
Design consistency	matplotlib defaults	Drifts every session	Design-system locked	Design-system locked
Security (XSS)	Sandbox exec risk	OWASP LLM05 exposure	Unrepresentable	Unrepresentable
Output tokens	~Python code	2k–4k / report	1.5k–3k / report	60–70% less than JSON
Schema evolution	Code-only	Free-for-all	Sprint per new widget	Add to component palette
PDF export	Native (already image)	Fragile (headless browser)	Custom renderer	Same DSL, static renderer
Debuggability	Opaque binary	Opaque DOM	Verbose JSON diff	Plain-text, diffable

Five Lessons from Production

Lesson 1: Constraints Are the Product, Not the Limitation

The Pydantic era taught us this lesson once. The Artifact era taught it to us a second time. Both architectures work for the same underlying reason: the agent doesn’t get to invent the visual layer. It picks from a vocabulary we control. The two architectures differ on format — typed JSON versus line-oriented DSL — but they agree on the contract: the agent describes intent, the renderer owns the pixels.

When we first proposed a whitelist of components, the obvious pushback was: “Won’t that limit what the agent can do?” Yes. That’s the entire point. Every dashboard the agent produces inherits CloudThinker’s design system for free — because it’s literally rendered by the same components the rest of the product uses. There is no drift because there is no vocabulary to drift within.

The ACM DIS 2025 GenUI Study found that 37 UX professionals, after a week with state-of-the-art GenUI tools, agreed “GenUI quality was not production-ready and would require further tweaking and polishing” — with accessibility and domain-specificity as the main complaints. Both are artifacts of Open-ended generation. In a Controlled pattern with a whitelist, accessibility is a property of the component (we audit LineChart once), and domain-specificity is the entire reason the vocabulary exists.

If the agent wants a new kind of component, we add it once. Not per prompt. Not per session. Once.

Lesson 2: Streaming Changes UX More Than You Think

Streaming insight

You can’t stream a PNG — it’s atomic. You can’t safely stream raw HTML — half-rendered markup is broken markup. You can stream a line-oriented DSL, because every line is a complete, parseable component. A dashboard that takes JSON 14.2 seconds to finish streams in OpenUI Lang in 4.9 seconds — not because the model is faster, but because the grammar was designed for progressive rendering from day one.

This is the lesson the Pydantic era couldn’t teach us. Constrained-decoding JSON is correct, but it’s a wall — the model commits nothing until the tree validates, and the user stares at a spinner. A line-oriented grammar streams differently from anything else. As each line of kpi = KPICard(...) arrives, the renderer can immediately show that card. The user sees a skeleton, then KPIs, then charts, then tables — progressively, in the order the agent generated them. Perceived latency drops more than actual latency does, because the user has something to look at from the second second onward.

You cannot do this with a PNG (it’s atomic). You cannot safely do this with raw HTML (half-streamed HTML is broken HTML). You cannot do it with JSON-based payloads either (the tree isn’t valid until the closing brace, and constrained decoding makes the model wait for it). You can do it with a grammar where each line is independently parseable. Streaming isn’t a feature you bolt on to the output format — it’s a constraint the output format has to satisfy from day one, and it’s why every serious GenUI project has ended up rethinking the payload shape.

Lesson 3: Token Economics Compound at Agent Scale

Per-request savings don’t matter at low volume. They matter enormously at agent volume. When the same agent generates many artifacts in a single long-running investigation — dashboards, tables, sub-reports, one-offs — the per-artifact token delta turns into a per-user cost delta, and the per-user delta turns into a unit-economics story.

The Pydantic era taught us that a strict schema is necessary but not sufficient. JSON costs you tokens you didn’t budget for: every widget repeats its discriminator, every layout repeats its grid math, every chart repeats its axis config. Published benchmarks across the GenUI category show DSL-based payloads running 60-67% fewer tokens than JSON-based alternatives for the body alone. But the bigger win isn’t the payload tokens — it’s the instruction tokens. Teaching the agent a small vocabulary is much shorter than teaching it “how to fill in our dashboard schema correctly.” The system prompt shrinks. The few-shot examples shrink. The agent has less to think about, so it thinks faster.

The hook numbers at the top of this post — 30-40 seconds down to around 10 seconds, ~$0.50 per report down to ~$0.08 — are the compounded effect of all of this. Smaller output, shorter system prompt, faster rendering, no waiting on constrained decoding to finish a JSON tree.

Lesson 4: One DSL, Many Surfaces

Same artifact body, multiple renderers:

Web: React components (Highcharts inside Card containers)
PDF: WeasyPrint-friendly HTML, for digest reports and audit exports
Future Slack / email: a straightforward mapping to Slack Block Kit or MJML templates

You cannot do this with matplotlib (one output: PNG). You cannot do this with hand-written React (tightly coupled to a browser runtime). The Pydantic era was halfway there — the JSON tree was renderer-agnostic in principle, but in practice the React renderer and the PDF renderer drifted because the schema was too rich to fully cover twice. The DSL is tight enough that both renderers stay honest.

This also turns out to be how we handle editing. A user says “make the chart bigger.” The agent doesn’t re-generate from scratch — it produces a diff against the existing DSL, and the renderer re-paints. The DSL is a first-class document, not a one-shot output.

Lesson 5: Debuggability Is a Superpower

Artifacts are stored as plain-text DSL in PostgreSQL. This has three properties we didn’t plan for but can’t live without:

Diffs. “Why did the agent pick a bar chart here instead of a line chart last week?” is a SELECT query followed by a text diff. The Pydantic era’s JSON dumps were technically diffable too, but the noise-to-signal ratio was bad — every diff was 80% positional reshuffling.
Replay. Loading a past artifact and re-rendering it costs nothing. No code execution, no sandbox, no model call. Useful for bug reports and regression tests.
Auditability. Every report is a human-readable, version-controllable document. For customers in regulated industries, this matters a lot.

Contrast: if your agent output is a PNG, good luck explaining to a compliance auditor why the chart looks the way it does.

What We’d Do Differently

The honest retrospective:

The first migration is the one you should expect. The most expensive lesson from the Pydantic era was assuming we’d get the format right on the first try. We knew matplotlib and HTML were dead ends and we picked the next-most-obvious thing — strict typed JSON via constrained decoding. It worked. It was also a stepping stone, not a destination. If you’re building a Generative UI system today, budget for at least one rewrite of the payload format. Preferably plan for it from the start.

Ship the DSL spec first, then the components. We built components opportunistically the second time around — BarChart when we needed a bar chart, GaugeChart when we needed a gauge. In hindsight we should have started by writing down the entire intended vocabulary as a locked spec, validated it against a handful of target reports, and then built components. The irony: we did exactly this with Pydantic schemas the first time. Doing it twice would have been smarter than doing it once perfectly.

Prompt engineering is most of the work. The DSL is trivial — a few dozen components, a few hundred lines of parser. What took the longest was teaching the agent when to use a KPICard versus a Callout, when a GaugeChart is better than a single-value metric, when to group cards into Stack("row") versus a vertical layout. The Thesys guidance on building Generative UI applications turned out to be more useful than we expected — the design patterns matter more than the framework.

Validation errors need to be LLM-friendly. Our first parser returned Python tracebacks to the agent when the DSL was malformed. The agent got confused by the tracebacks and hallucinated more errors trying to fix them. Now the parser returns structured hints — “You called LineChart without a labels array. Expected: LineChart(labels, series). Did you mean: LineChart(['Jan', 'Feb'], [Series('Cost', [100, 200])])?” Error messages are prompts. Treat them as such. (The Pydantic era got this right by accident — Pydantic’s structured error messages are already pretty good.)

Don’t skip the PDF renderer. Building the WeasyPrint HTML renderer in parallel with the React one forced us to keep the DSL truly declarative. Any component that secretly needed client-side JavaScript to render was a leak — and having a second renderer exposed every leak immediately. If you’re building a Generative UI system and you only target the browser, you’ll accrue hidden coupling you won’t notice until the first time you need to export.

What’s Next — From Snapshots to Live Workspaces

Today’s artifacts are honest read-only documents. The agent emits the DSL, the renderer paints it, and that’s where the conversation ends. You can look at the dashboard, but you can’t do anything to it — every follow-up question kicks off a fresh generation pass and a new artifact appears next to the old one. It works, but it treats the artifact like a printout rather than a workspace.

The next iteration we’re building flips that. Artifacts become stateful surfaces the user and the agent share, not snapshots the agent throws over the wall. Three concrete shifts:

Bidirectional artifacts. Click a region on the cost map and the agent drills in. Drag a date range on a line chart and the agent re-runs the underlying query against the new window. Hover a row in a service breakdown and the agent surfaces the related anomalies. The artifact stops being a destination and starts being an input — every interaction is an implicit prompt that the agent picks up on the next turn.

Editable artifacts via DSL diffs. “Make the time range 90 days” should not require a full regeneration. The agent emits a small DSL diff against the existing artifact, the renderer re-paints in place, and the audit log records the delta. Artifacts become version-controlled documents — you can branch them, replay them, and trace exactly which agent changed which panel and why.

Embedded controls as first-class DSL primitives. New components — Filter(), DateRange(), Toggle(), Slider() — that the agent can drop into an artifact and the renderer wires up to a re-run. The DSL graduates from a layout language to an interaction language, but the contract stays the same: the agent describes intent, the renderer owns the pixels and the events. The PDF renderer degrades each control gracefully — a DateRange() becomes a static caption in print, a button in Slack, a live picker in the browser. One body, many surfaces, all of them alive.

The deeper bet: Generative UI is going to look obvious in 18 months, the same way streaming chat responses look obvious now. The surface area of “what the agent produces” is going to keep getting smaller and more semantic, and the surface area of “what the product renders” is going to keep getting richer. The line between them is the contract that matters — and as Roger Wong put it, we’re heading toward interfaces that exist for a single moment, composed on demand, thrown away after.

A quick shout-out to the teams pushing Generative UI forward: Vercel for shipping the first widely-used streaming UI primitive, LangChain for the graph-native approach, Thesys for the category-defining work on C1, CopilotKit for the AG-UI protocol, Google for A2UI, and OpenUI for pushing DSL-based streaming further than anyone else. The ecosystem is better because of all of them — and the pattern is only going to get more prevalent from here.

Further reading: OpenUI Lang Benchmarks · CopilotKit: The Developer’s Guide to Generative UI in 2026 · OWASP LLM05: Improper Output Handling · ACM DIS 2025: The GenUI Study · Generative UI and the Ephemeral Interface

First published on cloudthinker.io on April 6, 2026.

Agentic infrastructure, entry 1: We Tried Every AI Sandbox. Then We Built Our Own.

Khai Trinh — Sat, 21 Mar 2026 00:00:00 GMT

Hosted sandboxes couldn’t reach our private APIs. Self-hosted ones needed dedicated servers. So we built our own.

One of our AI agents spent 40 minutes debugging a cloud networking issue. It installed diagnostic tools, traced network configurations, and correlated logs across multiple services. Three commands from the root cause — the kind of deep investigation that separates a useful agent from a chatbot.

Then the session timed out. Everything gone. The agent started over from zero, reinstalling the same tools, re-tracing the same logs, re-learning the same context. Forty minutes of compute burned twice.

The problem wasn’t just cost — it was the architecture: sandboxes that lived outside our network, couldn’t reach our internal APIs, required enterprise plans for basic features like private network access, and gave us no control over what our agents could access. When you’re running AI agents against production infrastructure, “it works but we can’t see what it’s doing” isn’t a feature — it’s a liability.

So we evaluated every option on the market, found an open-source project with the right foundation, added the production features it was missing, and open-sourced the result.

TL;DR — OpenSandbox on EKS: Self-hosted AI sandbox infrastructure with persistent filesystems, tiered pause/resume, network security controls, and security hardening — running on your own AWS account. Get started.

Choosing the Right Sandbox

We evaluated the main categories of AI sandbox options. Every category broke for different reasons.

Hosted Sandbox APIs

Managed infrastructure, fast cold starts, SDKs in every language — we had agents running in an afternoon. Then we tried to connect an agent to our internal Grafana instance. Nothing. The sandbox lived in the provider’s network — completely isolated from our private infrastructure. Internal APIs, private databases, monitoring dashboards — all unreachable. Connecting to our network required expensive enterprise tiers ($3K+/month), and even then you don’t control the underlying infrastructure.

The pricing model was also wrong for AI workloads. Per-second billing works for quick, short-lived tasks. AI agents are different — they run long sessions, install tools, explore systems, and iterate. You’re paying for compute the entire time, with no way to pause a sandbox to cheaper storage when it’s idle.

Some hosted sandboxes now offer basic outbound traffic filtering, but you’re configuring security rules through an API you don’t own. When an incident happens, you can’t inspect the rules, review traffic logs, or audit what your agent actually accessed.

Self-Hosted VM-Based Alternatives

The alternative camp offered something different: run sandboxes on your own infrastructure. Open source, self-hosted, full control. Exactly what we wanted — in theory.

In practice, these tools required dedicated physical servers or traditional VMs. They weren’t designed for Kubernetes — the orchestration platform we’d already invested years building around. That meant no automatic scaling, no cost-saving spot instances, and no integration with the infrastructure we already had.

The operational model was also limiting. Most VM-based solutions only support destroy-and-recreate — there’s no concept of pausing a sandbox and resuming it later with state intact. For AI agents that work on multi-day investigations, this is a dealbreaker. You either keep the VM running (expensive) or destroy it and lose everything (wasteful).

Found a Kubernetes-Native Foundation

After these dead ends, we found Alibaba’s OpenSandbox — an open-source project that got the fundamentals right.

It was Kubernetes-native from the ground up — designed for container orchestration, not a VM tool with a Kubernetes wrapper bolted on. It included a lifecycle management server, an execution engine inside each sandbox, and SDKs in five languages (Python, TypeScript, Java, C#, Go).

But honest assessment: it was a foundation, not a production system. For running on AWS at scale, critical features were missing:

No filesystem persistence — sandbox state didn’t survive restarts
No tiered pause/resume — no way to suspend sandboxes at different cost tiers
No network security beyond basic defaults — no outbound filtering, no cloud credential protection
Minimal base image — missing the programming languages, CLIs, and tools agents actually need
Security gaps — admin access, excessive permissions, weak isolation

The foundation was right. The production layer was ours to build.

Problems we kept hitting

No Private Connectivity. Hosted sandboxes live outside your network. Agents can’t reach internal APIs, private databases, or monitoring systems without ugly workarounds.
Expensive at Scale. Per-second billing compounds fast with long-running AI sessions. No way to pause idle sandboxes to cheaper storage tiers.
No Affordable Persistence. Hosted persistence requires expensive enterprise tiers. Self-hosted VMs offer destroy-and-recreate only. No tiered pause/resume to optimize idle costs.
Limited Network Visibility. Hosted sandboxes offer basic egress rules, but you can’t see or audit the traffic. Self-hosted gives you full visibility into what your agents access.

Architecture in 60 Seconds

SDK calls create Sandbox resources, managed by the controller into isolated containers with persistent filesystems.

The request flow is straightforward:

Call Sandbox.create() via any SDK (Python, TypeScript, Java, C#, Go)
FastAPI lifecycle server creates a Sandbox resource in Kubernetes
The controller creates an isolated container with persistent storage and network rules
A setup step installs the execution engine into the container
The startup script sets up the persistent filesystem and security restrictions
SDK communicates with the execution engine for code execution, file ops, and shell commands

The entire lifecycle — create, pause, resume, archive, terminate — is managed through standard Kubernetes resources. No extra orchestration layers. No external databases. Standard tools work for debugging.

In an upcoming post, we’ll do a full technical deep dive into filesystem persistence, controller design, network security, and storage tiering.

What We Added

Now that you’ve seen how the pieces fit together, here’s what we built on top of the open-source foundation.

1. Persistent Filesystem

Standard containers lose all changes when they restart. We added a persistent filesystem layer so every package install, config change, and downloaded file survives restarts. The agent installs tools once. They’re still there next week.

2. Tiered Pause & Resume

Not all sandboxes are equally active. Running sandboxes sit at full compute cost, but idle ones should cost almost nothing.

Tier	What’s kept	Resume Time	Cost (20GB)
Active	Full compute + disk	Instant	~$5/mo
Warm (paused)	Disk only	~5 seconds	~$1.60/mo
Cold (archived)	Snapshot only	~30 seconds	~$0.50/mo

Snapshots only store the data your agent actually wrote — not the full 20GB volume. A sandbox that used 10GB of its 20GB disk costs about $0.50/mo as a snapshot.

Cost per agent, 20GB EBS:

Storage	Cost	What it means
No Tiering	~$5/mo	Every sandbox stays at full cost, active or idle
3-Tier Storage	~$0.50/mo	Idle sandboxes archive to cold — 10x cheaper

Why this matters: Agents are idle most of the time — they only wake when an event fires (PR opened, issue created, scheduled task). Cold storage at $0.50/mo is the steady state, not the exception.

An agent that’s idle for a few hours gets paused to warm. Idle for days, archived to cold. Resume from any tier restores the full filesystem — every installed tool, every config file, every work-in-progress artifact.

3. Network Security & Filtering

Every sandbox gets its own network filtering layer. Unlike hosted solutions where you configure rules through an API you don’t control, OpenSandbox gives you full visibility — standard network policies you can audit, customize, and integrate with your existing security tooling.

4. Pre-Built Environment

Every sandbox comes with programming languages, cloud tools, monitoring integrations, and developer utilities pre-installed. No more spending the first 5 minutes of every session installing tools.

5. Security Hardening

Running untrusted AI-generated code requires a locked-down environment:

No admin access — agents run as unprivileged users
Privilege escalation blocked — no way for code to gain elevated permissions
Minimal permissions — only what’s needed during initial setup, then immediately restricted
Shared directories locked down — no hijacking temp files or shared paths

What we added to OpenSandbox

Persistent Filesystem. Install packages once, they survive pause/resume. No more cold environments.
Tiered Pause & Resume. Pause idle sandboxes to save ~68%. Resume in ~5s warm, ~30s cold. Storage drops to ~$0.50/mo.
Network Security & Filtering. Each sandbox gets its own network rules. Block access to internal services, cloud credentials, and sensitive endpoints.
Pre-Built Environment. Programming languages, cloud tools, monitoring integrations, and developer utilities — all pre-installed.
MCP Servers Included. Monitoring, database, and infrastructure integrations ready to go out of the box.
Security Hardening. No admin access, privilege escalation blocked, minimal permissions, shared directories locked down.

OpenSandbox at Scale

Here’s what OpenSandbox looks like running at scale on AWS:

Each AI agent gets its own isolated pod with persistent storage, pre-installed tools, and network filtering. Idle agents pause to save ~68%. Dormant agents archive to snapshots at ~$0.50/mo.

Every AI agent at CloudThinker runs inside its own isolated OpenSandbox environment:

What’s inside each sandbox (OpenSandbox Pod, per agent, persistent filesystem):

Category	Tools
Cloud CLIs	aws, gcloud, az
MCP Servers	grafana, elasticsearch, sonarqube, zabbix
Cluster & Git	kubectl, helm, git, gh
Dev Utilities	ripgrep, jq, curl, python3, node, bun

The typical lifecycle: an agent is assigned a task, creates or resumes a sandbox, works for minutes or hours, then goes idle. The system automatically pauses to warm storage. If the user comes back next week, the sandbox resumes from a snapshot in about 30 seconds — every tool still installed, every file still in place.

Here’s a concrete example. One agent was debugging an intermittent connection timeout between a cloud service and a database. Day one: installed diagnostic tools, traced the issue to a networking bug. Day two: the engineer asked the agent to continue. It resumed in 4 seconds with everything intact — captured data, scripts, partial analysis. No reinstallation. It identified the fix within an hour.

What We Learned

Persistence changes everything

One of our agents had been working for 11 hours — it had installed diagnostic tools, cloned repositories, and was correlating logs across multiple services. At 3 AM, the spot instance it was running on got terminated.

The system moved it to a new server. Every file, every tool, every piece of progress — intact. The agent didn’t even know it moved.

Without persistence, that’s 11 hours of wasted compute. With it, it’s a non-event. This is what convinced us persistence isn’t optional — it’s the foundation. Everything else we build assumes the sandbox survives failures.

Design for idle, not for active

Agents are event-driven — they activate when a PR opens or an issue is created, then sit idle. Without tiered storage, you pay full price for every sandbox whether it’s working or not.

With tiered storage, idle sandboxes drop from ~$5/mo to ~$0.50/mo — about 90% cheaper. At 200 sandboxes where 80% are idle, that’s the difference between ~$1,000/mo and ~$200/mo.

The insight: most infrastructure is designed for peak load. Agent infrastructure should be designed for idle.

Why Open Source

When AI runs code against production, you need to see the isolation layer. If an agent executes kubectl delete namespace production, the difference between “that’s fine, it was sandboxed” and a career-ending incident is the sandbox implementation. That implementation shouldn’t be a black box behind an API.

The ecosystem also needs a shared standard. Every AI platform is building proprietary sandbox infrastructure — duplicated engineering that doesn’t interoperate. We’d rather compete on what agents accomplish than on container plumbing.

And we need help. Specifically:

Firecracker / gVisor — stronger isolation using secure container runtimes
Local development — running sandboxes on your laptop without a full cloud cluster

If any of these problems interest you, the issue tracker has tagged good-first-issue items for each.

What’s Next

Secure container runtimes — gVisor, Kata Containers, and Firecracker for deeper isolation at the operating system level
Tiered cold storage — automatic archival for sandboxes that haven’t been accessed in weeks, driven by usage metrics
Snapshot pre-warming — predictive resume that restores sandboxes before the user asks, based on usage patterns

Get Involved

GitHub: cloudthinker-ai/opensandbox-on-eks
Getting Started: Quick start guide

Building AI agents and fighting sandbox problems? Open an issue — we read every one.

Coming Up Next

Part 2: 1,000 AI Agents, One Cluster — The Architecture That Holds

The full technical deep dive: how OverlayFS keeps agent work alive across node failures, how tiered storage cuts costs by 80-90%, and why blocking one IP address prevents an entire class of privilege escalation attacks.

First published on cloudthinker.io on March 21, 2026.

Agentic memory, entry 1: Agent Memory Meets Graph: Introducing MemGraph — Long-Term Memory for AI Cloud Agents

Henry Bui — Wed, 18 Mar 2026 00:00:00 GMT

Your AI agent brilliantly diagnosed a Lambda-to-RDS connection storm last Tuesday. It identified the missing connection pooling, recommended the fix, and walked your team through implementation.

On Wednesday, the exact same pattern appeared in a different service. The agent started from zero.

This is the fundamental problem with AI agents today. They’re stateless. Every conversation is a blank slate. The hard-won operational knowledge your team builds through months of incidents, optimizations, and architectural decisions — the agent forgets all of it the moment the session ends.

We built MemGraph to fix this.

The Memory Problem No One Talks About

Most AI agent platforms treat memory as an afterthought — a simple vector store that retrieves relevant documents. This approach has three fatal flaws:

No Relationships. Every memory is an island. A flat document store can’t express that “before you resize an EKS node group, you must understand node affinity rules.” There’s no concept of prerequisite knowledge, conflicting approaches, or superseded procedures.

No Evolution. Knowledge grows stale. That gp2 resize procedure from 6 months ago sits alongside the current gp3 migration guide with equal weight. Outdated knowledge is worse than no knowledge — it leads agents to confidently give wrong answers.

No Signal. All memories are equal. A procedure validated 50 times in production ranks the same as one used once and never confirmed. There’s no concept of earned trust, reinforcement, or contradiction.

Traditional RAG is a filing cabinet. You can find documents, but the cabinet doesn’t know which documents are related, which are outdated, or which ones actually worked in practice. MemGraph is an experienced engineer’s brain — it connects knowledge, prioritizes what’s proven, and quietly deprecates what’s no longer relevant.

Introducing MemGraph: Memory as a Knowledge Graph

MemGraph treats every piece of operational knowledge not as a document, but as a node in a graph — connected to other knowledge through typed, meaningful relationships.

Five Memory Types

Not all knowledge is the same. MemGraph classifies memories into five distinct types, each optimized for different retrieval patterns:

Procedural — Step-by-step workflows and tool sequences
Factual — Rules, constraints, and best practices
Episodic — Success stories with reasoning
Incident — Root cause analysis from real incidents
Developer Pattern — Team-specific habits and preferences

Procedural Memory — The How-To Library

Reusable step-by-step workflows that solved real problems. These are the agent’s playbook — battle-tested recipes that worked in your specific environment.

Task: Resize EKS node group with zero downtime
Solution:
  1. Cordon existing nodes (prevent new pod scheduling)
  2. Create new node group with target instance type
  3. Wait for new nodes to reach Ready state
  4. Drain old nodes (graceful pod eviction with PDB respect)
  5. Verify all pods rescheduled on new nodes
  6. Delete old node group
Context: Production EKS clusters with PodDisruptionBudgets configured

Factual Memory — The Rulebook

Hard constraints, best practices, and organizational rules. These prevent agents from suggesting non-compliant configurations, even if they’d be cheaper or simpler.

Task: RDS instance configuration requirements
Solution: Production RDS instances in us-east-1 MUST use Multi-AZ deployment.
          Single-AZ is only permitted for dev/staging environments.
          Minimum backup retention: 7 days for prod, 1 day for non-prod.
Context: Compliance requirement from SOC2 audit (2025-Q3)

Episodic Memory — The War Stories

Successful resolutions with the reasoning behind why they worked — enabling agents to apply similar reasoning to new problems.

Task: Reduce EBS costs for data processing workloads
Solution: Migrated 47 gp2 volumes to gp3 with custom IOPS/throughput tuning.
          Result: 42% cost reduction ($8,400/month savings) with 15% better throughput.
Reasoning: gp2 IOPS scales linearly with volume size, forcing over-provisioning.
           gp3 decouples IOPS from size, so you only pay for what you use.

Incident Memory — The Post-Mortems

Root cause analysis from real incidents, turned into reusable diagnostic patterns. When a similar pattern emerges, the agent immediately recognizes it and applies the proven fix instead of diagnosing from scratch.

Task: Lambda-to-RDS connection exhaustion (P1 incident)
Root Cause: Lambda functions creating new DB connections per invocation.
            Under load (500 concurrent), exceeded RDS max_connections (150).
Solution: Implemented RDS Proxy with connection pooling.
          Set max_connections_percent=70 to reserve headroom.
          Added CloudWatch alarm on DatabaseConnections metric.

Developer Pattern Memory — The Team Playbook

Team-specific preferences, conventions, and tooling choices. These let agents adapt recommendations to match how your team actually works — not generic best practices, but your best practices.

Task: Infrastructure deployment preferences for Platform Team
Solution: Team uses Terraform modules exclusively (no raw CloudFormation).
          Naming convention: {env}-{service}-{resource} (e.g., prod-api-rds).
          Prefers blue-green over canary for stateful services.

Seven Relationship Types — The “Graph” in MemGraph

The real power isn’t in individual memories — it’s in how they connect. MemGraph supports seven typed relationships between memory nodes:

Relationship	Meaning	Example
prerequisite	Must understand this before applying the memory	“Before resizing EKS nodes → understand PodDisruptionBudgets”
enables	This knowledge unlocks a more advanced capability	“Understanding VPC peering → enables cross-account access patterns”
alternative	Different approach to the same problem	“Spot instances vs. Reserved Instances for batch workloads”
extends	Adds depth or capability to existing knowledge	“Basic S3 lifecycle rules → extended with Intelligent Tiering”
supersedes	Replaces outdated knowledge	“gp3 migration guide supersedes old gp2 resize procedure”
conflicts	Contradicts — cannot apply both	“Spot instances conflict with guaranteed SLA requirements”
related	General semantic connection	“RDS optimization relates to connection pooling best practices”

Why relationships matter: When an agent retrieves a memory about EKS node resizing, MemGraph automatically surfaces prerequisite knowledge about PodDisruptionBudgets. It flags that the old EC2-based resize method has been superseded. It warns that the spot instance approach conflicts with this workload’s SLA requirements. The agent doesn’t just get an answer — it gets the full context to apply it correctly.

The Evolution Pipeline — Memories That Learn

Most memory systems only append. Every new piece of knowledge gets added to an ever-growing pile. Over time, the pile becomes noise.

MemGraph curates. Every new memory goes through an LLM-driven evolution pipeline that decides how it should integrate with existing knowledge.

1. Extract — After each conversation, an LLM analyzes the dialogue and extracts 0 to 5 reusable patterns — not raw chat logs, but distilled operational knowledge: the methodology that worked, the reasoning behind it, the context where it applies.

2. Search — For each extracted memory, MemGraph searches the existing knowledge graph for related memories using hybrid search — combining exact keyword matching (BM25) with semantic similarity (vector embeddings) through Reciprocal Rank Fusion.

3. Evolve — An LLM analyzes the new memory against search results and makes one of three decisions:

STORE — This is genuinely new knowledge. Add it to the graph and build typed relationships to existing memories.
COMBINE — This overlaps with something we already know. Merge them into a single, stronger, more complete memory.
DISCARD — We already know this. Don’t pollute the graph with duplicates.

4. Connect — For stored or combined memories, the LLM determines which existing memories have meaningful relationships — prerequisites, alternatives, conflicts — and builds typed edges. Keywords from related memories are enriched into the new node.

This is the architectural edge. Over time, MemGraph doesn’t just grow — it gets sharper. Combined memories become more complete. Superseded memories get flagged. Validated procedures get reinforcement signals. The knowledge graph converges toward a refined, high-signal representation of your team’s operational expertise.

Confidence Signals — Earned Trust, Not Static Scores

Every memory in MemGraph carries dynamic confidence signals that evolve over time:

Access Count — How many times this memory has been retrieved. Frequently accessed memories are likely valuable.
Reinforcement — How many times new evidence has confirmed this memory. Validated knowledge ranks higher.
Contradiction — How many times new evidence has conflicted with this memory. Flags unreliable knowledge.
Supersession — Whether this memory has been replaced by a newer, better version. Prevents serving outdated advice.

A procedure that’s been reinforced 50 times gets a 5-50% score boost in search results. A memory that’s been contradicted multiple times gets flagged for review. This is earned trust — the graph learns what works in your environment over time.

Hybrid Search — Finding the Right Memory at the Right Time

When an agent needs operational knowledge, MemGraph doesn’t just do a simple text search. It combines three retrieval strategies:

Keyword Search (BM25) — Sparse vector matching that finds memories containing exact terms from your query. When you search for “RDS connection timeout”, it finds memories that specifically mention those words. Best for known error messages, service names, specific configurations.

Semantic Search (Embeddings) — Dense vector similarity that finds memories conceptually related, even if they use different terminology. A search for “database connection issues” finds memories about “connection pooling” and “RDS Proxy setup.” Best for exploratory queries and “how do I…” questions.

Graph Traversal — When a relevant memory is found, MemGraph follows its prerequisite and enables edges to surface the full context. Finding “RDS Proxy Setup” automatically pulls in “VPC Security Group Configuration” (prerequisite) and “Multi-AZ Failover Optimization” (enables). Best for complete context retrieval.

The three strategies are combined using Reciprocal Rank Fusion (RRF), with an additional boost for memories with high reinforcement counts. The result: agents get the most relevant, battle-tested knowledge — with full context — every time.

Real-World Impact

Without MemGraph

You: @alex We're seeing connection timeouts on the orders-api RDS instance

Alex: I'll investigate the RDS connection timeout issue.
      Let me check the current metrics...

      [Runs CloudWatch queries from scratch]
      [Analyzes connection count patterns]
      [Researches possible causes]
      [Eventually identifies connection pooling issue]

      Time to diagnosis: 12 minutes
      Same diagnosis your team did 3 months ago
      Knowledge lost after session ends

With MemGraph

You: @alex We're seeing connection timeouts on the orders-api RDS instance

Alex: MemGraph match: This pattern matches a P1 incident from 3 months ago
      (Lambda-to-RDS connection exhaustion, confidence: high, reinforced 12x)

      Prerequisite: Checking if RDS Proxy is configured...
      It's not — this is the same root cause.

      Recommended fix (proven procedure):
      1. Deploy RDS Proxy with connection pooling (max_connections_percent=70)
      2. Update Lambda security groups for proxy access
      3. Add CloudWatch alarm on DatabaseConnections metric

      Time to diagnosis: 8 seconds
      Battle-tested fix, validated 12 times across your infrastructure
      Related: Also surfacing VPC security group prereqs

Use Cases in Production

Incident Memory — From 12-minute diagnosis to 8-second recognition. When similar incident patterns emerge, MemGraph immediately surfaces the proven root cause and fix — no re-diagnosis needed. The agent learns from every P1 your team resolves.

Cross-Conversation Context — Daily summaries and conversation memories let agents understand ongoing decisions: “The team agreed to freeze spot instance changes until after the SOC2 audit.” No one has to re-explain context.

Personalized Operations — Per-user memory files store communication preferences, priorities, and constraints. The agent knows this SRE prefers Terraform, that security lead requires compliance-first framing, and the CTO wants executive summaries.

Team Pattern Recognition — MemGraph learns that your platform team uses Helm charts over raw manifests, prefers blue-green over canary, and has specific naming conventions. The agent adapts its suggestions to fit — not fight — your existing practices.

Under the Hood — Architecture

MemGraph is built on four storage backends, each optimized for its role:

Layer	Backend	What It Stores	Why This Backend
Knowledge Graph	Qdrant (vector DB)	Memory nodes with typed relationships, dense + sparse vectors	Hybrid search (keyword + semantic) with metadata filtering
Conversation History	Qdrant (separate collection)	Topic summaries, key decisions, action items	Semantic retrieval across time windows (1-365 days)
Personal Context	S3/MinIO	Per-user preference files, per-workspace daily summaries	Durable, structured markdown with Redis caching (600s TTL)
Performance Cache	Redis	Search results, deduplication tracking	Sub-millisecond reads, automatic expiration

Technical Details

Embedding Model: Amazon Bedrock Cohere Embed v4 — generates both dense vectors (semantic similarity) and sparse vectors (BM25-style keyword matching) in a single API call.

Search Algorithm: Reciprocal Rank Fusion (RRF) combines keyword and semantic search results, with a configurable reinforcement boost (5-50%) for validated memories.

Processing Pipeline: Fully async via Celery task queue. Memory extraction runs on high-priority workers with 300-second timeouts. Search results are cached in Redis for 1 hour to avoid redundant computation.

Deduplication: Content-hash-based dedup (MD5 normalized) with 24-hour TTL prevents the same knowledge from being stored multiple times across concurrent agent sessions.

Graph Traversal: When include_related=true, search results are enriched by following prerequisite and enables edges — surfacing the full context chain without requiring additional queries.

The Compound Effect

Here’s what makes MemGraph fundamentally different from traditional memory systems:

Every conversation makes your agents smarter.

Unlike fine-tuning — which is expensive, batch-oriented, and risky — MemGraph evolves in real-time, per-workspace, with full auditability. You can inspect every memory node, trace every relationship, and understand exactly why the agent made a recommendation.

The graph starts sparse. After a month of operations, prerequisite chains emerge. After three months, the evolution pipeline is actively combining related memories and superseding outdated ones. After six months, your agents carry the operational context of a senior engineer who’s been on your team from the start.

That’s the compound effect. Not just remembering — connecting, validating, and refining knowledge with every conversation.

MemGraph is available today for all CloudThinker workspaces. Your agents are already learning from every conversation, building a knowledge graph unique to your infrastructure, your team’s practices, and your operational history.

First published on cloudthinker.io on March 18, 2026.

Agentic system design, entry 1: CloudThinker Agentic Orchestration and Context Optimization

Henry Bui — Mon, 24 Nov 2025 00:00:00 GMT

1. Introduction

We’ve all built chatbots. They are great at answering questions, but terrible at doing actual work. The industry is shifting from reactive chatbots to autonomous agentic systems, but this transition isn’t free.

When we started building CloudThinker, we wanted more than a chat interface. We wanted an AI that could wake up, analyze a skyrocketing cloud bill, fix the database config causing it, and send us a report—all without us typing a word.

But moving from “talking” to “doing” is hard. Single-agent architectures eventually hit a wall with long-running workflows. They can’t distribute specialized tasks, parallelize operations, or manage the exponential token growth that comes with complex reasoning. If you’re not careful, they burn through $50 of tokens in ten minutes just to tell you “I don’t know.”

We built CloudThinker to solve these scale problems. By implementing novel context optimization techniques—prompt caching, asynchronous context compaction, and parallel tool calling—we achieved 80-95% cost reduction, 7x faster task completion, and 85% latency reduction compared to our baseline.

Here is the architecture we built to achieve those numbers, and the production lessons we learned moving from a fragile prototype to a system that actually works.

2. Multi-Agent Architecture Deep Dive

Multi-agent orchestration unlocks the potential of specialized AI systems, but success hinges on coordination strategy. How do you prevent chaos when multiple agents work simultaneously?

2.1 The Coordination Verdict

Let’s be clear: Always start with a single agent.

Multi-agent orchestration is cool, but it’s a nightmare to debug. We didn’t switch to a team of agents because it was trendy. We switched because our single agent hit a ceiling. It couldn’t be an expert in Kubernetes and billing and security all at once. It started making mistakes. Only then did we accept the “complexity tax” of building a team.

The Evolution: From Single-Agent to Supervisor

Our architecture didn’t start with specialists. It evolved because we looked at the metrics. We compared three fundamental approaches:

Pattern	Coordination Model	Verdict
Single-Agent	No coordination needed	❌ The Ceiling: Great for simple tasks, fails hard on complex ones.
Network (Peer-to-Peer)	Distributed consensus	❌ The Trap: Like a committee meeting with no agenda. Lots of talking, no decisions.
Supervisor	Centralized coordinator	✅ The Solution: Clear ownership. One boss, many workers.

Single-Agent (The Baseline): Always start here. We ran a single generalist agent until it started failing on complex, multi-domain tasks. Don’t pay the “coordination tax” until you have to.
Network/Peer-to-Peer (The “Chaos” Trap): We tried letting agents talk directly to each other. It was a mess. Without a boss in the room, they argued in infinite loops about who should check the database.
The Winner: Supervisor Pattern: We settled on the Supervisor Pattern (Anna - General Manager). It gave us the reliability we needed: one agent whose only job is to make sure the work gets done.

We standardized on the Supervisor Pattern (Anna - General Manager). This provides the “Production-Ready” reliability we needed: clear ownership, explicit routing, and a central point for state management.

2.2 Choosing the Right Supervisor Variant

Once we decided on a Supervisor, we had to pick a management style.

Variant	When to Use	Trade-offs
Flat Supervisor	Small teams (2-10)	✅ Simple, reliable ← CloudThinker uses this
Supervisor (as tools)	Delegation	⚠️ Less autonomy, tighter coupling
Hierarchical	Large Orgs	⚠️ Good for scale, bad for latency. Adds middle management.
Custom Graph	Complex dependencies	❌ Flexible, but debugging it is miserable.

Flat Supervisor (CloudThinker’s Choice): We stuck with the simplest option. We have distinct domains (Compute, Database, Security), so a single flat layer is enough.
Hierarchical: Only use this if you’re building a massive system. If you need a “Support Team” that handles tickets without ever bothering the main manager, this makes sense. Otherwise, it’s just extra layers.
Custom Graph: Avoid this unless you hate yourself. In production, we prefer boring and predictable over “flexible” and broken.

Production Insight: Start with the Flat Supervisor. It incurs the lowest “coordination tax” while providing sufficient separation of concerns. Only move to Hierarchical if you need to scale beyond 10 agents or require strict “Transfer” patterns where sub-teams operate completely independently.

2.3 Routing and Activation: Optimistic Routing with Fail-Safes

In theory, the Supervisor should manage everything. In reality, that’s slow and expensive. Why ask the manager for permission to check the time?

We use Optimistic Routing—we assume the specialist can handle it, but we keep a safety net.

Default to the Specialist (Fast Path): If you ask “Show running instances,” we send you straight to Alex (Cloud Engineer). No meeting with the boss required. Zero orchestration overhead.
Contextual Continuity (Sticky Sessions): If you’re already talking to Tony about a database issue, we keep you there. It’s annoying to be transferred back to the main menu every time you reply.
The Escalation Safety Net: This is the “fail-safe.” If you ask Alex (the cloud guy) “Why is my RDS slow?”, he doesn’t try to guess. He says, “That’s not my job,” and escalates back to Anna (Supervisor). She then assigns it to the right person. Complex tasks never fail silently; they just get escalated.

Why this matters: This approach cuts latency by 40-50% for most tasks. We keep the speed of a single agent but the power of a full team when we need it.

2.4 Agent Communication Protocol

If you think getting three humans to agree on a lunch spot is hard, try getting three AI agents to debug a database.

Without strict rules, multi-agent chats turn into “context pollution”—agents confusing each other with irrelevant data until the whole system crashes.

We solve this with a Group Chat Protocol that enforces two simple rules:

Explicit Targeting: You can’t just “talk.” You must address someone (@alex, @anna). If you don’t, the message is rejected.
Structured Handoffs: You can’t just dump a 50-page log file into the chat. You have to summarize it first.

Within this framework, we use two patterns to move work around: Delegation (The “Boomerang”) and Transfer (The “Handoff”).

Delegation: The “Boomerang” Pattern

Complex infrastructure workflows require multiple specialized agents working in sequence, but naive implementations create two critical problems: (1) exponential token growth when each agent inherits all previous context, and (2) coordination failures when handoffs lose critical task state. Delegation solves this through structured task assignment where supervisors retain accountability while workers execute specialized operations.

Consider a multi-stage AWS cost optimization workflow: (1) Alex (Cloud Engineer) gathers EC2 spending data, (2) Anna (General Manager) analyzes spending patterns and identifies database cost anomalies, (3) Tony (Database Engineer) investigates RDS configuration and proposes optimizations. This can’t be a direct transfer—Anna needs Alex’s cost data to provide strategic context to Tony, but Tony shouldn’t inherit Alex’s verbose CloudWatch API outputs (potentially 50K+ tokens). Delegation enables this through structured handoffs: Anna delegates to Alex → Alex executes and returns summarized results to group chat → Anna takes results, analyzes, then delegates to Tony → Tony executes with only Anna’s analysis context, not Alex’s raw outputs.

Delegation operates through the group chat tool with explicit continuation markers. The critical optimization: Tony receives Anna’s 200-token delegation message with strategic context (“no compute correlation”), not Alex’s 50K-token CloudWatch dumps. Anna maintains accountability—she’s waiting for Tony’s results to compile the final optimization plan.

Token Economics and State Management

Each agent maintains isolated message history—when Agent A delegates to Agent B, only the handoff message appears in Agent B’s context, not Agent A’s tool outputs. With this context isolation approach:

Agent A: 50K tokens → produces 200-token summary
Agent B: 200 tokens (A’s summary) + 30K (B’s work) = 30.2K tokens → produces 200-token summary
Agent C: 200 tokens (B’s summary) + 20K (C’s work) = 20.2K tokens
Total: 100.4K tokens processed (56% reduction)

In production CloudThinker workflows, delegation with context isolation achieves 50-70% token reduction compared to shared-context implementations, directly reducing inference costs and latency.

Trade-offs and Production Considerations

Delegation introduces coordination overhead—each handoff requires supervisor reasoning about which specialist to activate and what context to provide. In our metrics, delegation adds around 5 seconds per handoff for supervisor decision-making, acceptable for complex workflows but unnecessary for simple single-specialist tasks. The most common anti-pattern is delegating too early: if the supervisor hasn’t completed its analysis phase, delegation becomes premature—“@alex go check EC2 costs” without specifying what patterns to look for forces unnecessary back-and-forth. The supervisor must provide sufficient task framing before delegating. Error handling adds further complexity—when a worker fails mid-delegation chain, the supervising agent must decide: retry with the same specialist, escalate to a different specialist, or abort the workflow. Production systems require explicit error handling policies; our implementation uses a 3-retry policy with exponential backoff before escalating to the user.

Transfer: The “Handoff” Pattern

Not all multi-agent workflows require supervisor oversight. Transfer provides complete ownership handoff for simple, domain-specific queries—the supervisor exits, the specialist takes full control, and all subsequent user interactions happen directly with the specialist. The choice hinges on two questions: (1) Does the task require multi-specialist coordination? (2) Will the supervisor need the results for subsequent analysis? If both answers are “no,” transfer is optimal.

Transfer applies to single-specialist queries: “What’s our PostgreSQL version?” or “Show current Kubernetes pods.” Delegation handles multi-stage workflows: “Why did costs spike?” (requires coordination) or “Investigate API slowness” (requires multiple specialists).

Transfer eliminates supervisor overhead by archiving the supervisor’s state after handoff—no callbacks needed. Production metrics show cost reduction and 50% latency reduction compared to delegation (10.2K tokens → 7.2K tokens, ~4-6s → ~2-3s). However, the most common failure is premature transfer: supervisors must complete decomposition before handing off.

❌ "App has high latency" → "@alex investigate" (too vague).
✅ "API response times 200ms→1.8s at 14:00 UTC" → "@alex investigate infrastructure causes" (sufficient context). Transfer only when the specialist has sufficient context to execute independently.

3. Context Optimization Techniques

Multi-agent orchestration solves coordination challenges but introduces a new problem: token explosion. Four techniques form our optimization strategy—prompt caching, asynchronous context compaction, tool consolidation, and parallel tool calling.

3.1 Prompt Caching

The ratios between chatbot and agentic systems are dramatically different. Chatbot systems typically have a 3:1 input-to-output token ratio, while agentic systems can reach 100:1 ratios. This is because agentic systems need to reason, plan, and execute tasks through multiple tool calls—often 50+ per task. Input token costs dominate total expenses, making prompt caching critical for production viability.

Prompt caching (also called KV cache) allows LLMs to reuse previously computed context across API calls, delivering two critical benefits: cost reduction and latency improvement. Most implementations charge a premium for cache writes but deliver significant discounts on cache reads (often 80-90% cost reduction) and up to 85% latency reduction on cached requests. Cache hits also dramatically reduce time-to-first-token (TTFT)—the model skips reprocessing cached context and immediately begins generating output. After just a few API calls reusing the same context, the economics shift dramatically in your favor.

The Cache Breakpoint Challenge

The rule is simple: Cache everything static. Keep dynamic stuff at the end.

But “static” is trickier than it looks. As the Manus team discovered, a single timestamp in the wrong place can invalidate your entire cache. We treat Cache Hit Rate as a top-level KPI. If it drops below 90%, we treat it like a production outage.

Three-Tier Objective Prompt Strategy

Following the Manus team’s insight that objective prompts should be placed outside cache boundaries, we use a “Three-Tier” architecture to maximize cache reuse while preserving turn-by-turn adaptability:

The diagram illustrates the critical optimization: the cache checkpoint moves forward after each completed turn. At Step N, the agent generates Action 1 and receives Observation 1. At Step N+1, these completed interactions move behind the checkpoint—now cached and reusable. The Objective (containing live context like team composition, active resources, and current plan) remains after the checkpoint, regenerated each turn with updated information. By Step N+2, both previous action-observation pairs are cached. This “moving breakpoint” strategy ensures all historical work gets cached while only the dynamic objective block is reprocessed each turn. Compare this to naive implementations that modify system prompts or insert content mid-history—those approaches invalidate the entire cache on every API call.

The three-tier structure:

Tier 1 - Static System Prompt: Tool schemas and core behavioral rules. Cache write on first call, then cache hits forever (95%+ hit rate across conversations)
Tier 2 - Conversation History: Actions and Observations accumulate as the agent works. The cache checkpoint moves forward after each completed turn, preserving all historical context
Tier 3 - Dynamic Objectives: Always placed AFTER the checkpoint. Contains live context (team composition, connected resources, active plan, memory retrieval results). Never cached—regenerated each turn with current state

Key Insight: By keeping dynamic content at the end, everything upstream remains cacheable. As conversations grow longer, the cache absorbs the majority of tokens while only the small objective block is reprocessed per turn—the foundation of the 80-95% cost reduction.

3.2 Asynchronous Context Compaction

Caching saves money, but you eventually run out of space. Even with a 200k context window, a multi-hour infrastructure analysis can fill the buffer with tool outputs and reasoning traces.

The standard industry solution is “summarization,” but the way most teams implement it destroys both User Experience (UX) and accuracy.

The Failure of Synchronous Summarization

We initially ran summarization synchronously: when the window hit 90%, the agent would pause, call a separate “Summarizer Agent” to compress the history, and then resume.

This failed for two reasons:

The “Please Wait” Problem: Users were left staring at a spinner for 45 seconds in the middle of a debug session while the system “cleaned up.” It killed the flow.
The “External Summarizer” Gap: Our benchmarks showed that handing the history to a generic third-party summarizer agent resulted in performance degradation on subsequent tasks. The external agent lacked the implicit reasoning state of the active worker, resulting in generic summaries that stripped out critical technical nuances.

The Fix: Asynchronous Self-Summarization

We moved to an Asynchronous Context Compaction model that solves both problems.

The 70% Trigger: We don’t wait until the cliff edge. When context usage hits 70%, we trigger a background job.
Non-Blocking Execution: The user doesn’t see this happen. The main agent continues responding to new queries using the full (un-compacted) context, maintaining zero latency.
Hot-Swapping: When the background task completes, we carefully splice the state. We replace the old history (up to the trigger point) with the new summary, but we keep the new messages (generated while the background task was running) raw and untouched.

Why Self-Summarization Wins: Crucially, we don’t spawn a new “Summarizer” persona. We ask the active agent itself to “Summarize your work so far” in that background thread. Because the agent summarizes its own conversation, it intuitively preserves the details relevant to its current goal, rather than creating a generic recap.

This approach eliminated the “maintenance pause” entirely and improved long-context task completion rates by keeping recent context raw and immediate.

The “Cache Trap” in Summarization

Even with asynchronous execution, you still have to pay for the tokens to generate the summary. Here is where most teams lose money. When it’s time to summarize, the instinct is to change the system prompt to: “You are a summarizer.”

This is a trap. Changing the system prompt breaks the cache for the entire history. You effectively pay full price to re-read 100k tokens just to throw them away.

The Fix: Append, Don’t Replace

The fix is simple: Don’t touch the system prompt. Just append a final message: “Summarize the conversation above.” This keeps the 100k tokens in the cache (90% discount).

Claude Sonnet 4.5 pricing: $3/MTok input, $15/MTok output, $0.30/MTok cache read

❌ NAIVE: Create new system prompt with summarization instructions → breaks cache

{
    "system": [
        {
            "type": "text",
            "text": "You are a summarizer agent..."  # New system prompt (~500 tokens)
        },
    ],
    "messages": [
        # Old messages to summarize (100K tokens)
        # CACHE BROKEN - must reprocess as uncached input!
        {"role": "user", "content": "Message 1..."},
        {"role": "assistant", "content": "Response 1..."},
        ...,
        {"role": "user", "content": "Message 50..."}
    ]
}

Cost breakdown: 500 tokens uncached ($0.0015) + 100K uncached ($0.300) + 2K output ($0.030) = $0.3315

✅ OPTIMIZED: Preserve original system prompt + append summarization instruction → preserves cache

{
    "system": [
        {
            "type": "text",
            "text": "You are Anna..."  # Original system prompt (~8K tokens, already cached)
        },
        {"cachePoint": {"type": "default"}}  # Cache hit!
    ],
    "messages": [
        # Old messages to summarize (100K tokens)
        {"role": "user", "content": "Message 1..."},
        {"role": "assistant", "content": "Response 1..."},
        ...,
        {"role": "user", "content": "Message 50..."},
        {"cachePoint": {"type": "default"}},  # Cache all old messages

        # Summarization instruction at the END (~500 tokens, not cached)
        {
            "role": "user",
            "content": `Summarize the conversation above.
            Extract: key decisions, technical analysis, errors, current state.`,
        },
    ],

}

Cost breakdown: 8K cached ($0.0024) + 100K cached ($0.030) + 0.5K uncached ($0.0015) + 2K output ($0.030) = $0.0639

Savings: 80.7% cost reduction ($0.3315 → $0.0639)

Trade-offs and Recursive Summarization: Summarization loses some conversational nuance and fine-grained detail. We mitigate this through comprehensive summary templates that preserve technical decisions, error patterns, and user feedback—ensuring agents maintain continuity across compaction boundaries.

For extremely long sessions, we apply recursive summarization: existing summaries get included in subsequent passes, creating hierarchical compression where older context becomes progressively condensed while recent context remains detailed.

3.3 Tool Consolidation

Anthropic recently introduced the Model Context Protocol (MCP) as a standard for connecting agents to data. It’s a massive step forward—standardizing how agents connect to everything from Slack to Postgres.

The “Tool Pollution” Paradox

MCP makes it easy to connect tools. Too easy. Suddenly, your agent has access to 500 tools. If you dump 500 tool definitions into the context window, you leave no room for reasoning. The agent spends 90% of its brainpower just parsing the API list, or worse, hallucinates parameters because the context is flooded.

The “Code Execution” Pivot (and why we use it sparingly)

To solve this, Anthropic also proposed Code Execution with MCP, suggesting agents write code to “discover” tools dynamically. Instead of listing tools in the system prompt, you give the agent an execution environment and let it import tool definitions from a file tree.

The Reliability Trade-off

While progressive disclosure is efficient, we found that for our use case, keeping critical tools visible is essential. In CloudThinker, reliability is our primary goal. We think that if an agent doesn’t explicitly see a “Restart Database” tool in its system prompt, it is less likely to reason about using it as part of a complex solution.

The CloudThinker Solution: Consolidated Tools + JIT Schemas

We chose a pragmatic middle ground: Consolidate the interface, hide the manual.

Traditional CRUD agent design creates five separate tools (recommendation_list, recommendation_get, recommendation_create, recommendation_update, recommendation_delete). We merge these into a single interface.

# Minimal description + schema externalized to get_instruction
@tool(description=MINIMAL_RECOMMENDATION_TOOL_DESCRIPTION)
async def recommendation(
    command: Literal["get_instruction", "get_all", "delete", "create", "update"],
    recommendation_ids: list[UUID] | None = None,
    recommendations: dict | None = None, # Generic dict instead of typed schema
) -> str:

The “Just-in-Time” Schema

Notice the get_instruction command? That’s our secret weapon.

Instead of stuffing the full schema into the system prompt (where you pay for it on every token), we hide it behind this command.

Visibility: The agent sees the recommendation tool. It knows the capability exists (unlike Code Execution).
Efficiency: It doesn’t pay for the parameter definitions until it needs them.

If the agent forgets how to update a recommendation, it calls recommendation(command="get_instruction") to get the manual. This moves documentation from “Always Loaded” (expensive) to “On Demand” (cheap), maintaining reliability without the token bloat.

3.4 Parallel Tool Calling

Early LLM agent architectures were fundamentally sequential. The ReAct pattern (Reasoning and Acting), introduced in 2022, defined the standard approach: the LLM calls one tool, analyzes the outcome, reasons about the next step, then executes another tool call. This sequential workflow was by design—models weren’t trained to handle multiple simultaneous operations, and frameworks enforced one-tool-at-a-time execution to maintain reliability.

The landscape shifted dramatically in 2024. When Anthropic announced Claude 3’s tool use general availability in May 2024, parallel tool calling was initially unsupported. However, as foundation models became more capable, providers began training them to handle concurrent operations. Modern Claude models now support calling multiple tools simultaneously within a single response—a critical evolution for complex workflows requiring independent operations.

Enabling Parallel Tool Calling

Model capability alone isn’t sufficient—you must explicitly instruct agents to leverage parallelism. Modern models default to sequential execution unless prompted otherwise. As Anthropic’s documentation recommends: “For maximum efficiency, whenever you need to perform multiple independent operations, invoke all relevant tools simultaneously rather than sequentially.” This single instruction can reduce multi-tool workflow latency by 3-5x compared to sequential execution.

In CloudThinker, parallel tool calling emerges naturally during plan execution. When an agent marks a plan step completed and transitions to the next task, it evaluates whether multiple operations can run concurrently. For example, completing “gather cost data” might trigger parallel steps for “analyze compute patterns,” “review storage utilization,” and “audit network traffic”—all executing simultaneously rather than sequentially, reducing multi-step workflows from minutes to seconds.

This isn’t just about speed (though it cuts latency by 3-5x). It’s about money. Every round trip to the model costs you tokens. By batching 5 tool calls into one request, we cut out 4 round trips of context processing. That simple change drove a 30-40% cost reduction for our complex workflows.

4. Conclusion

We learned the hard way that multi-agent systems fail predictably—and succeed through disciplined optimization.

The supervisor-worker pattern proved essential for operational clarity—predictable failures, traceable routing, and observable trade-offs. But remember the golden rule:

Complexity is a choice.

Start with a single agent. Add prompt caching. Instrument everything. Only when metrics demand it, evolve to a supervisor pattern. Scale when metrics justify complexity.

First published on cloudthinker.io on November 24, 2025.