# CloudThinker Engineering: all journeys and entries --- --- # Agentic system design Question: How should we structure an agent system so it stays fast, cheap and reliable as tasks get longer and use more tools? Status: ongoing. Started 2025-11-24. Last entry 2026-04-15. 2 entries, 32 min read, 11k tokens. How we build the agent system itself: how agents hand work to each other, how prompts are laid out, and how tool calls are run. We measure latency, tokens and cost on every change. ## What we think now Revision 2, 2026-04-15. 1. Start with one agent. Move to a flat supervisor with workers only when the numbers ask for it; a flat supervisor is enough up to about 10 agents. 2. Keep each agent's context separate and hand over short summaries (about 200 tokens), not whole transcripts. This cut tokens by 50 to 70%. 3. Prefer transfer over delegation when the worker can answer the user itself. It halved latency and cut one task from 10.2K to 7.2K tokens. 4. Lay out prompts for the cache. Static content first, then a checkpoint that moves down each turn, then the objective last. A cache hit rate under 90% is treated as an outage. 5. Compact the context at 70% full, by appending the summarize instruction instead of swapping the system prompt. That keeps the cache and was 80.7% cheaper ($0.3315 to $0.0639). 6. Start each tool as soon as its call has finished streaming, not when the whole message ends. About 50% faster on the median production trace, 1.20 to 1.50 times faster than plain parallel calls in the benchmark. 7. Do not start a tool early if it is not idempotent, takes under 50 ms, or needs another tool's result. ## Where we changed our minds - 2026-04-15: ~~Run all tool calls in parallel once the model's message is complete.~~ Start each tool as soon as its own call is complete, while the model is still streaming. (entry 2: https://engineering.cloudthinker.io/journeys/agentic-system-design/02.md) ## Open questions - Does hiding tool schemas the agent does not need hurt reliability? - At what size does a flat supervisor stop being enough? - Can a tool that is not idempotent be started early safely? ## Entries 1. [CloudThinker Agentic Orchestration and Context Optimization](https://engineering.cloudthinker.io/journeys/agentic-system-design/01.md) (2025-11-24, start, 20 min, 7.0k tokens) 2. [Eager Tool Calling: How We Cut Agent Latency by 50% on Long Tool Chains](https://engineering.cloudthinker.io/journeys/agentic-system-design/02.md) (2026-04-15, experiment, 12 min, 4.0k tokens) Web version: https://engineering.cloudthinker.io/journeys/agentic-system-design/ --- # CloudThinker Agentic Orchestration and Context Optimization Journey: [Agentic system design](https://engineering.cloudthinker.io/journeys/agentic-system-design.md). Entry 1 of 2. Posted 2025-11-24 by Henry Bui. Type: start. ## 1. Introduction We've all built chatbots. They are great at answering questions, but terrible at doing actual work. The industry is shifting from reactive chatbots to autonomous agentic systems, but this transition isn't free. When we started building CloudThinker, we wanted more than a chat interface. We wanted an AI that could wake up, analyze a skyrocketing cloud bill, fix the database config causing it, and send us a report—all without us typing a word. But moving from _"talking"_ to _"doing"_ is hard. Single-agent architectures eventually hit a wall with long-running workflows. They can't distribute specialized tasks, parallelize operations, or manage the exponential token growth that comes with complex reasoning. If you're not careful, they burn through $50 of tokens in ten minutes just to tell you _"I don't know."_ We built CloudThinker to solve these scale problems. By implementing novel context optimization techniques—prompt caching, asynchronous context compaction, and parallel tool calling—we achieved **80-95% cost reduction**, **7x faster task completion**, and **85% latency reduction** compared to our baseline. Here is the architecture we built to achieve those numbers, and the production lessons we learned moving from a fragile prototype to a system that actually works. --- ## 2. Multi-Agent Architecture Deep Dive Multi-agent orchestration unlocks the potential of specialized AI systems, but success hinges on coordination strategy. How do you prevent chaos when multiple agents work simultaneously? ### 2.1 The Coordination Verdict Let's be clear: **Always start with a single agent.** Multi-agent orchestration is cool, but it's a nightmare to debug. We didn't switch to a team of agents because it was trendy. We switched because our single agent hit a ceiling. It couldn't be an expert in Kubernetes _and_ billing _and_ security all at once. It started making mistakes. Only then did we accept the _"complexity tax"_ of building a team. **The Evolution: From Single-Agent to Supervisor** Our architecture didn't start with specialists. It evolved because we looked at the metrics. We compared three fundamental approaches: | Pattern | Coordination Model | Verdict | | :------------------------- | :---------------------- | :--------------------------------------------------------------------------------------- | | **Single-Agent** | No coordination needed | ❌ **The Ceiling**: Great for simple tasks, fails hard on complex ones. | | **Network (Peer-to-Peer)** | Distributed consensus | ❌ **The Trap**: Like a committee meeting with no agenda. Lots of talking, no decisions. | | **Supervisor** | Centralized coordinator | ✅ **The Solution**: Clear ownership. One boss, many workers. | - **Single-Agent (The Baseline)**: Always start here. We ran a single generalist agent until it started failing on complex, multi-domain tasks. Don't pay the _"coordination tax"_ until you have to. - **Network/Peer-to-Peer (The "Chaos" Trap)**: We tried letting agents talk directly to each other. It was a mess. Without a boss in the room, they argued in infinite loops about who should check the database. - **The Winner: Supervisor Pattern**: We settled on the **Supervisor Pattern** (Anna - General Manager). It gave us the reliability we needed: one agent whose only job is to make sure the work gets done. We standardized on the **Supervisor Pattern** (Anna - General Manager). This provides the "Production-Ready" reliability we needed: clear ownership, explicit routing, and a central point for state management. ### 2.2 Choosing the Right Supervisor Variant Once we decided on a Supervisor, we had to pick a management style. | Variant | When to Use | Trade-offs | | :------------------------ | :------------------- | :---------------------------------------------------------- | | **Flat Supervisor** | Small teams (2-10) | ✅ **Simple, reliable** ← **CloudThinker uses this** | | **Supervisor (as tools)** | Delegation | ⚠️ Less autonomy, tighter coupling | | **Hierarchical** | Large Orgs | ⚠️ Good for scale, bad for latency. Adds middle management. | | **Custom Graph** | Complex dependencies | ❌ Flexible, but debugging it is miserable. | - **Flat Supervisor (CloudThinker's Choice)**: We stuck with the simplest option. We have distinct domains (Compute, Database, Security), so a single flat layer is enough. - **Hierarchical**: Only use this if you're building a massive system. If you need a _"Support Team"_ that handles tickets without ever bothering the main manager, this makes sense. Otherwise, it's just extra layers. - **Custom Graph**: Avoid this unless you hate yourself. In production, we prefer boring and predictable over _"flexible"_ and broken. **Production Insight**: Start with the **Flat Supervisor**. It incurs the lowest _"coordination tax"_ while providing sufficient separation of concerns. Only move to Hierarchical if you need to scale beyond 10 agents or require strict _"Transfer"_ patterns where sub-teams operate completely independently. ### 2.3 Routing and Activation: Optimistic Routing with Fail-Safes In theory, the Supervisor should manage everything. In reality, that's slow and expensive. Why ask the manager for permission to check the time? We use **Optimistic Routing**—we assume the specialist can handle it, but we keep a safety net. - **Default to the Specialist (Fast Path)**: If you ask _"Show running instances_," we send you straight to **Alex (Cloud Engineer)**. No meeting with the boss required. Zero orchestration overhead. - **Contextual Continuity (Sticky Sessions)**: If you're already talking to Tony about a database issue, we keep you there. It's annoying to be transferred back to the main menu every time you reply. - **The Escalation Safety Net**: This is the _"fail-safe."_ If you ask Alex (the cloud guy) _"Why is my RDS slow?"_, he doesn't try to guess. He says, _"That's not my job,"_ and escalates back to **Anna (Supervisor)**. She then assigns it to the right person. Complex tasks never fail silently; they just get escalated. **Why this matters**: This approach cuts latency by 40-50% for most tasks. We keep the speed of a single agent but the power of a full team when we need it. ### 2.4 Agent Communication Protocol If you think getting three humans to agree on a lunch spot is hard, try getting three AI agents to debug a database. Without strict rules, multi-agent chats turn into _"context pollution"_—agents confusing each other with irrelevant data until the whole system crashes. We solve this with a **Group Chat Protocol** that enforces two simple rules: 1. **Explicit Targeting**: You can't just _"talk."_ You must address someone (`@alex`, `@anna`). If you don't, the message is rejected. 2. **Structured Handoffs**: You can't just dump a 50-page log file into the chat. You have to summarize it first. Within this framework, we use two patterns to move work around: **Delegation** (The _"Boomerang"_) and **Transfer** (The _"Handoff"_). **Delegation: The "Boomerang" Pattern** ![Delegation pattern: over time, the supervisor delegates a task through the group chat, worker 1 takes it and posts an update, the supervisor takes that and delegates to worker 2, worker 2 posts an update, and the supervisor takes it and responds to the user](https://engineering.cloudthinker.io/diagrams/agentic-system-design/delegation-pattern.svg "wide") Complex infrastructure workflows require multiple specialized agents working in sequence, but naive implementations create two critical problems: (1) **exponential token growth** when each agent inherits all previous context, and (2) **coordination failures** when handoffs lose critical task state. Delegation solves this through structured task assignment where supervisors retain accountability while workers execute specialized operations. Consider a multi-stage AWS cost optimization workflow: (1) Alex (Cloud Engineer) gathers EC2 spending data, (2) Anna (General Manager) analyzes spending patterns and identifies database cost anomalies, (3) Tony (Database Engineer) investigates RDS configuration and proposes optimizations. This can't be a direct transfer—Anna needs Alex's cost data to provide strategic context to Tony, but Tony shouldn't inherit Alex's verbose CloudWatch API outputs (potentially 50K+ tokens). Delegation enables this through structured handoffs: Anna delegates to Alex → Alex executes and returns summarized results to group chat → Anna takes results, analyzes, then delegates to Tony → Tony executes with only Anna's analysis context, not Alex's raw outputs. Delegation operates through the group chat tool with explicit continuation markers. The critical optimization: Tony receives Anna's 200-token delegation message with strategic context (_"no compute correlation"_), not Alex's 50K-token CloudWatch dumps. Anna maintains accountability—she's waiting for Tony's results to compile the final optimization plan. **Token Economics and State Management** Each agent maintains isolated message history—when Agent A delegates to Agent B, only the handoff message appears in Agent B's context, not Agent A's tool outputs. With this context isolation approach: - Agent A: 50K tokens → produces 200-token summary - Agent B: 200 tokens (A's summary) + 30K (B's work) = 30.2K tokens → produces 200-token summary - Agent C: 200 tokens (B's summary) + 20K (C's work) = 20.2K tokens - **Total: 100.4K tokens processed (56% reduction)** In production CloudThinker workflows, delegation with context isolation achieves **50-70% token reduction** compared to shared-context implementations, directly reducing inference costs and latency. **Trade-offs and Production Considerations** Delegation introduces coordination overhead—each handoff requires supervisor reasoning about which specialist to activate and what context to provide. In our metrics, delegation adds around 5 seconds per handoff for supervisor decision-making, acceptable for complex workflows but unnecessary for simple single-specialist tasks. The most common anti-pattern is delegating too early: if the supervisor hasn't completed its analysis phase, delegation becomes premature—_"@alex go check EC2 costs"_ without specifying what patterns to look for forces unnecessary back-and-forth. The supervisor must provide sufficient task framing before delegating. Error handling adds further complexity—when a worker fails mid-delegation chain, the supervising agent must decide: retry with the same specialist, escalate to a different specialist, or abort the workflow. Production systems require explicit error handling policies; our implementation uses a 3-retry policy with exponential backoff before escalating to the user. **Transfer: The "Handoff" Pattern** ![Transfer pattern: over time, the supervisor transfers the task through the group chat, worker 1 takes it and responds to the group chat and directly to the user, with no return to the supervisor](https://engineering.cloudthinker.io/diagrams/agentic-system-design/transfer-pattern.svg) Not all multi-agent workflows require supervisor oversight. Transfer provides complete ownership handoff for simple, domain-specific queries—the supervisor exits, the specialist takes full control, and all subsequent user interactions happen directly with the specialist. The choice hinges on two questions: (1) Does the task require multi-specialist coordination? (2) Will the supervisor need the results for subsequent analysis? If both answers are _"no,"_ transfer is optimal. Transfer applies to single-specialist queries: _"What's our PostgreSQL version?"_ or _"Show current Kubernetes pods."_ Delegation handles multi-stage workflows: _"Why did costs spike?"_ (requires coordination) or _"Investigate API slowness"_ (requires multiple specialists). Transfer eliminates supervisor overhead by archiving the supervisor's state after handoff—no callbacks needed. Production metrics show **cost reduction** and **50% latency reduction** compared to delegation (10.2K tokens → 7.2K tokens, ~4-6s → ~2-3s). However, the most common failure is premature transfer: supervisors must complete decomposition before handing off. - ❌ `"App has high latency" → "@alex investigate"` (too vague). - ✅ `"API response times 200ms→1.8s at 14:00 UTC" → "@alex investigate infrastructure causes"` (sufficient context). **Transfer only when the specialist has sufficient context to execute independently.** ## 3. Context Optimization Techniques Multi-agent orchestration solves coordination challenges but introduces a new problem: token explosion. Four techniques form our optimization strategy—prompt caching, asynchronous context compaction, tool consolidation, and parallel tool calling. ### 3.1 Prompt Caching The ratios between chatbot and agentic systems are dramatically different. Chatbot systems typically have a 3:1 input-to-output token ratio, while agentic systems can reach 100:1 ratios. This is because agentic systems need to reason, plan, and execute tasks through multiple tool calls—often 50+ per task. Input token costs dominate total expenses, making prompt caching critical for production viability. Prompt caching (also called KV cache) allows LLMs to reuse previously computed context across API calls, delivering two critical benefits: cost reduction and latency improvement. Most implementations charge a premium for cache writes but deliver significant discounts on cache reads (often 80-90% cost reduction) and up to 85% latency reduction on cached requests. Cache hits also dramatically reduce time-to-first-token (TTFT)—the model skips reprocessing cached context and immediately begins generating output. After just a few API calls reusing the same context, the economics shift dramatically in your favor. **The Cache Breakpoint Challenge** The rule is simple: **Cache everything static. Keep dynamic stuff at the end.** But _"static"_ is trickier than it looks. As the [Manus team discovered](https://manus.im/blog/Context-Engineering-for-AI-Agents-Lessons-from-Building-Manus), a single timestamp in the wrong place can invalidate your entire cache. We treat **Cache Hit Rate** as a top-level KPI. If it drops below 90%, we treat it like a production outage. **Three-Tier Objective Prompt Strategy** Following the Manus team's insight that objective prompts should be placed outside cache boundaries, we use a "Three-Tier" architecture to maximize cache reuse while preserving turn-by-turn adaptability: ![Three-tier prompt caching strategy: context at steps N, N+1 and N+2. Tool schemas and system prompt sit at the top, then completed actions and observations, then the cache checkpoint, then the objective. Each turn the checkpoint moves down past the newest action and observation, so earlier blocks are cache hits and only the newest pair is a cache write](https://engineering.cloudthinker.io/diagrams/agentic-system-design/prompt-caching-three-tier.svg "wide") The diagram illustrates the critical optimization: the cache checkpoint moves forward after each completed turn. At Step N, the agent generates Action 1 and receives Observation 1. At Step N+1, these completed interactions move behind the checkpoint—now cached and reusable. The Objective (containing live context like team composition, active resources, and current plan) remains after the checkpoint, regenerated each turn with updated information. By Step N+2, both previous action-observation pairs are cached. This _"moving breakpoint"_ strategy ensures all historical work gets cached while only the dynamic objective block is reprocessed each turn. Compare this to naive implementations that modify system prompts or insert content mid-history—those approaches invalidate the entire cache on every API call. The three-tier structure: - **Tier 1 - Static System Prompt**: Tool schemas and core behavioral rules. Cache write on first call, then cache hits forever (95%+ hit rate across conversations) - **Tier 2 - Conversation History**: Actions and Observations accumulate as the agent works. The cache checkpoint moves forward after each completed turn, preserving all historical context - **Tier 3 - Dynamic Objectives**: Always placed AFTER the checkpoint. Contains live context (team composition, connected resources, active plan, memory retrieval results). Never cached—regenerated each turn with current state **Key Insight:** By keeping dynamic content at the end, everything upstream remains cacheable. As conversations grow longer, the cache absorbs the majority of tokens while only the small objective block is reprocessed per turn—the foundation of the 80-95% cost reduction. ### 3.2 Asynchronous Context Compaction Caching saves money, but you eventually run out of space. Even with a 200k context window, a multi-hour infrastructure analysis can fill the buffer with tool outputs and reasoning traces. The standard industry solution is _"summarization,"_ but the way most teams implement it destroys both User Experience (UX) and accuracy. **The Failure of Synchronous Summarization** We initially ran summarization synchronously: when the window hit 90%, the agent would pause, call a separate "Summarizer Agent" to compress the history, and then resume. This failed for two reasons: 1. **The "Please Wait" Problem**: Users were left staring at a spinner for 45 seconds in the middle of a debug session while the system _"cleaned up."_ It killed the flow. 2. **The "External Summarizer" Gap**: Our benchmarks showed that handing the history to a generic third-party summarizer agent resulted in **performance degradation** on subsequent tasks. The external agent lacked the implicit reasoning state of the active worker, resulting in generic summaries that stripped out critical technical nuances. **The Fix: Asynchronous Self-Summarization** We moved to an **Asynchronous Context Compaction** model that solves both problems. 1. **The 70% Trigger**: We don't wait until the cliff edge. When context usage hits 70%, we trigger a background job. 2. **Non-Blocking Execution**: The user doesn't see this happen. The main agent continues responding to new queries using the full (un-compacted) context, maintaining zero latency. 3. **Hot-Swapping**: When the background task completes, we carefully splice the state. We replace the _old_ history (up to the trigger point) with the new summary, but we keep the _new_ messages (generated while the background task was running) raw and untouched. **Why Self-Summarization Wins**: Crucially, we don't spawn a new _"Summarizer"_ persona. We ask the _active agent itself_ to "Summarize your work so far" in that background thread. Because the agent summarizes its own conversation, it intuitively preserves the details relevant to its current goal, rather than creating a generic recap. This approach eliminated the _"maintenance pause"_ entirely and improved long-context task completion rates by keeping recent context raw and immediate. **The "Cache Trap" in Summarization** Even with asynchronous execution, you still have to pay for the tokens to generate the summary. Here is where most teams lose money. When it's time to summarize, the instinct is to change the system prompt to: _"You are a summarizer."_ **This is a trap.** Changing the system prompt breaks the cache for the entire history. You effectively pay full price to re-read 100k tokens just to throw them away. **The Fix: Append, Don't Replace** The fix is simple: **Don't touch the system prompt.** Just append a final message: _"Summarize the conversation above."_ This keeps the 100k tokens in the cache (90% discount). **Claude Sonnet 4.5 pricing:** $3/MTok input, $15/MTok output, $0.30/MTok cache read **❌ NAIVE: Create new system prompt with summarization instructions → breaks cache** ```python { "system": [ { "type": "text", "text": "You are a summarizer agent..." # New system prompt (~500 tokens) }, ], "messages": [ # Old messages to summarize (100K tokens) # CACHE BROKEN - must reprocess as uncached input! {"role": "user", "content": "Message 1..."}, {"role": "assistant", "content": "Response 1..."}, ..., {"role": "user", "content": "Message 50..."} ] } ``` **Cost breakdown:** 500 tokens uncached ($0.0015) + 100K uncached ($0.300) + 2K output ($0.030) = **$0.3315** **✅ OPTIMIZED: Preserve original system prompt + append summarization instruction → preserves cache** ```python { "system": [ { "type": "text", "text": "You are Anna..." # Original system prompt (~8K tokens, already cached) }, {"cachePoint": {"type": "default"}} # Cache hit! ], "messages": [ # Old messages to summarize (100K tokens) {"role": "user", "content": "Message 1..."}, {"role": "assistant", "content": "Response 1..."}, ..., {"role": "user", "content": "Message 50..."}, {"cachePoint": {"type": "default"}}, # Cache all old messages # Summarization instruction at the END (~500 tokens, not cached) { "role": "user", "content": `Summarize the conversation above. Extract: key decisions, technical analysis, errors, current state.`, }, ], } ``` **Cost breakdown:** 8K cached ($0.0024) + 100K cached ($0.030) + 0.5K uncached ($0.0015) + 2K output ($0.030) = **$0.0639** **Savings:** 80.7% cost reduction ($0.3315 → $0.0639) **Trade-offs and Recursive Summarization**: Summarization loses some conversational nuance and fine-grained detail. We mitigate this through comprehensive summary templates that preserve technical decisions, error patterns, and user feedback—ensuring agents maintain continuity across compaction boundaries. For extremely long sessions, we apply recursive summarization: existing summaries get included in subsequent passes, creating hierarchical compression where older context becomes progressively condensed while recent context remains detailed. ### 3.3 Tool Consolidation Anthropic recently introduced the [Model Context Protocol (MCP)](https://www.anthropic.com/news/model-context-protocol) as a standard for connecting agents to data. It's a massive step forward—standardizing how agents connect to everything from Slack to Postgres. **The "Tool Pollution" Paradox** MCP makes it easy to connect tools. Too easy. Suddenly, your agent has access to 500 tools. If you dump 500 tool definitions into the context window, you leave no room for reasoning. The agent spends 90% of its brainpower just parsing the API list, or worse, hallucinates parameters because the context is flooded. **The "Code Execution" Pivot (and why we use it sparingly)** To solve this, Anthropic also proposed [Code Execution with MCP](https://www.anthropic.com/engineering/code-execution-with-mcp), suggesting agents write code to _"discover"_ tools dynamically. Instead of listing tools in the system prompt, you give the agent an execution environment and let it import tool definitions from a file tree. **The Reliability Trade-off** While progressive disclosure is efficient, we found that for our use case, keeping critical tools visible is essential. In CloudThinker, reliability is our primary goal. We think that if an agent doesn't explicitly see a _"Restart Database"_ tool in its system prompt, it is less likely to reason about using it as part of a complex solution. **The CloudThinker Solution: Consolidated Tools + JIT Schemas** We chose a pragmatic middle ground: **Consolidate the interface, hide the manual.** Traditional CRUD agent design creates five separate tools (`recommendation_list`, `recommendation_get`, `recommendation_create`, `recommendation_update`, `recommendation_delete`). We merge these into a single interface. ```python # Minimal description + schema externalized to get_instruction @tool(description=MINIMAL_RECOMMENDATION_TOOL_DESCRIPTION) async def recommendation( command: Literal["get_instruction", "get_all", "delete", "create", "update"], recommendation_ids: list[UUID] | None = None, recommendations: dict | None = None, # Generic dict instead of typed schema ) -> str: ``` **The "Just-in-Time" Schema** Notice the `get_instruction` command? That's our secret weapon. Instead of stuffing the full schema into the system prompt (where you pay for it on every token), we hide it behind this command. 1. **Visibility**: The agent _sees_ the `recommendation` tool. It knows the capability exists (unlike Code Execution). 2. **Efficiency**: It doesn't pay for the parameter definitions until it needs them. If the agent forgets how to update a recommendation, it calls `recommendation(command="get_instruction")` to get the manual. This moves documentation from _"Always Loaded"_ (expensive) to _"On Demand"_ (cheap), maintaining reliability without the token bloat. ### 3.4 Parallel Tool Calling Early LLM agent architectures were fundamentally sequential. The ReAct pattern (Reasoning and Acting), introduced in 2022, defined the standard approach: the LLM calls one tool, analyzes the outcome, reasons about the next step, then executes another tool call. This sequential workflow was by design—models weren't trained to handle multiple simultaneous operations, and frameworks enforced one-tool-at-a-time execution to maintain reliability. The landscape shifted dramatically in 2024. When Anthropic announced Claude 3's tool use general availability in May 2024, parallel tool calling was initially unsupported. However, as foundation models became more capable, providers began training them to handle concurrent operations. Modern Claude models now support calling multiple tools simultaneously within a single response—a critical evolution for complex workflows requiring independent operations. **Enabling Parallel Tool Calling** Model capability alone isn't sufficient—you must explicitly instruct agents to leverage parallelism. Modern models default to sequential execution unless prompted otherwise. As Anthropic's documentation recommends: ["For maximum efficiency, whenever you need to perform multiple independent operations, invoke all relevant tools simultaneously rather than sequentially."](https://docs.claude.com/en/docs/agents-and-tools/tool-use/implement-tool-use#maximizing-parallel-tool-use) This single instruction can reduce multi-tool workflow latency by 3-5x compared to sequential execution. In CloudThinker, parallel tool calling emerges naturally during plan execution. When an agent marks a plan step completed and transitions to the next task, it evaluates whether multiple operations can run concurrently. For example, completing _"gather cost data"_ might trigger parallel steps for _"analyze compute patterns,"_ _"review storage utilization,"_ and _"audit network traffic"_—all executing simultaneously rather than sequentially, reducing multi-step workflows from minutes to seconds. This isn't just about speed (though it cuts latency by 3-5x). It's about money. Every round trip to the model costs you tokens. By batching 5 tool calls into one request, we cut out 4 round trips of context processing. That simple change drove a **30-40% cost reduction** for our complex workflows. ## 4. Conclusion We learned the hard way that multi-agent systems fail predictably—and succeed through disciplined optimization. The supervisor-worker pattern proved essential for operational clarity—predictable failures, traceable routing, and observable trade-offs. But remember the golden rule: **Complexity is a choice.** Start with a single agent. Add prompt caching. Instrument everything. Only when metrics demand it, evolve to a supervisor pattern. Scale when metrics justify complexity. *First published on [cloudthinker.io](https://cloudthinker.io/blogs/cloudthinker-agentic-orchestration-and-context-optimization) on November 24, 2025.* --- Next: [Eager Tool Calling: How We Cut Agent Latency by 50% on Long Tool Chains](https://engineering.cloudthinker.io/journeys/agentic-system-design/02.md) Web version: https://engineering.cloudthinker.io/journeys/agentic-system-design/01/ --- # Eager Tool Calling: How We Cut Agent Latency by 50% on Long Tool Chains Journey: [Agentic system design](https://engineering.cloudthinker.io/journeys/agentic-system-design.md). Entry 2 of 2. Posted 2026-04-15 by Henry Bui. Type: experiment. ## 1. Introduction — The Latency Nobody Was Measuring An eight-tool agent task in our staging environment clocked in at twenty-four seconds. The model was fast. The individual tools were fast. The wall clock was slow. Where did the time go? We dropped the trace into a flame graph and the answer was embarrassing. For the first four seconds the model streamed tokens while every tool sat idle. Then the model finished, fired all eight tools in parallel, and spent another twenty seconds waiting for them. Two phases, stacked end to end, with no overlap between them. We rewrote that codepath. The same request now finishes in roughly five seconds — the four-second stream and the tool execution happen in the same four seconds instead of consecutively. Across a representative sample of production traces, median end-to-end task completion is **50% faster**, and longer tool chains see proportionally larger wins. The technique is called **eager tool calling** (internally we call it _tool-call pipelining_). Think of it as CPU instruction pipelining for agents: just as a modern CPU doesn't wait for one instruction to retire before decoding the next, an eager runtime doesn't wait for the model to finish before starting the tools it has already described. The idea is small. The engineering was not. Here is what it is, how it works, the production bugs that taught us what not to do, and the numbers we measured. --- ## 2. The Problem: Where Agent Latency Actually Hides The common intuition is that parallel tool calling already solved this. It didn't. Parallel tool calling is necessary but not sufficient. ### The classic agent loop A single agent turn looks like this: 1. Model reasons over the context and begins streaming a response. 2. The response contains one or more `tool_use` blocks. 3. Runtime waits for `message_stop`. 4. Runtime executes the tools. 5. Runtime collects the `tool_result` blocks and starts the next turn. Total wall clock for one turn = **stream time + tool time**, added serially. The tool layer is a passive consumer that refuses to start work until the producer has completely finished. ### The parallel tool calling half-fix Modern APIs — Anthropic, OpenAI, Bedrock — let the model emit multiple `tool_use` blocks in a single assistant message, and mature runtimes run those blocks concurrently. That change moves the tool phase from _sum_ of tool durations to _max_ of tool durations. Genuinely valuable. On an eight-tool turn with tools averaging 2.5 seconds, parallel tool calling takes ~2.5 seconds for the tool phase instead of 20. Big win. But the stream phase still happens first. The tools still wait for `message_stop`. If the model takes four seconds to emit its response, those four seconds are pure dead time on the tool side. ### What we wanted Tools that start running the instant the model finishes emitting them, _while the model continues streaming the rest of the message._ Not tools-parallel-with-tools. **Tools-parallel-with-generation.** Parallel tool calling overlaps tools with each other. Eager tool calling overlaps tools with the model itself. --- ## 3. Three Eras of Tool Calling | Era | Concurrency | When tools start | Wall clock | | --------------------- | ---------------------------------------------- | --------------------------------------------- | -------------------------- | | **Sequential** | None | After each prior tool finishes | Σ(stream + all tools) | | **Parallel** | Tools with tools | After `message_stop` | stream + max(tool) | | **Eager (pipelined)** | Tools with tools **and** tools with generation | The moment each tool block finishes streaming | **max(stream, max(tool))** | The move from sequential to parallel collapses one dimension of latency. The move from parallel to eager collapses the other. Time goes from sum to max in both directions. ![Wall clock comparison of parallel and eager tool calling with the same 4.0s stream and three 2.0s tools: parallel starts all tools at message_stop and finishes at 6.0s; eager starts tools A, B and C at 0.5s, 1.5s and 2.5s during the stream and finishes at 4.5s, 1.5s saved, 1.33× faster](https://engineering.cloudthinker.io/diagrams/agentic-system-design/eager-vs-parallel-timeline.svg "wide") --- ## 4. How Eager Tool Calling Works ### The key insight — the seal Each `tool_use` block the model emits has a stable `tool_call_id`. During streaming, the runtime sees chunks arrive one at a time. Most chunks carry a `tool_call_id`; partial argument chunks carry only an `index`. Here is the observation that makes eager tool calling possible: **the moment a chunk arrives with a `tool_call_id` different from the previous one, the previous tool call is definitionally complete.** The model has moved on. Its arguments are fully accumulated. It is safe to execute _right now_, even though the assistant message is still streaming. We call that transition the **seal event**. A tool is _sealed_ the instant the model starts streaming a different tool — or when the message itself ends. ### How it works in practice Each chunk that arrives with a new tool ID is proof the previous tool is fully assembled — we fire it immediately, while the model keeps streaming the rest of the message. ### Visualizing the overlap In a correct eager execution, the tool lanes are _not_ end-to-end. They _overlap_: ```text stream : [==================================] tool A : [=========] ← fires when B's id arrives tool B : [=========] ← fires when C's id arrives; runs concurrent with A tool C : [=========] ← fires at message_stop; runs concurrent with A + B ``` Drop a vertical line at the 50% mark and it cuts through all three bars. That is the signature of eager tool calling. If your "eager" diagram shows tools end-to-end, you have implemented sequential tool calling under a different name. ![Seal mechanism: stream chunks id=A, id=A, then id=B arrive; the seal detector sees prev_id != new_id, marks tool A complete and fires on_tool_call_sealed(A); the executor pool starts tool A while chunk B keeps streaming, with tools B and C queued](https://engineering.cloudthinker.io/diagrams/agentic-system-design/seal-mechanism.svg "wide") --- ## 5. Inside CloudThinker's Stream Handler The dispatch logic is straightforward to describe, even if the implementation required care. We track the most-recent tool call ID seen in the stream. The moment a new ID arrives, the previous tool is complete — we dispatch it immediately to a background worker pool without waiting for the rest of the message. That's the entire seal trigger: one state comparison per chunk. Partial argument chunks don't carry an ID — they only carry a positional index. So we maintain a small index-to-ID map that lets us route each partial chunk back to the right accumulation buffer, keeping the arguments intact even when multiple tools are streaming interleaved. The dispatch is fire-and-forget by design. If a background worker raises an exception, it's caught and surfaced as an error `tool_result` on the next turn. The stream reader never sees it, and the other tools keep running. ![Stream handler architecture in five stages: provider stream of SSE chunks, StreamHandler with SealDetector, on_tool_call_sealed callback, ExecutorPool of asyncio tool tasks, and a ToolMessage buffer for the next turn; the stream-to-callback boundary is fire-and-forget and the callback-to-executor boundary is a cancellation scope](https://engineering.cloudthinker.io/diagrams/agentic-system-design/stream-handler-architecture.svg "wide") --- ## 6. Production Lessons — What We Learned the Hard Way The first surprise was **tool retraction**. The model occasionally emits a tool call mid-stream, reconsiders, and replaces it. Rare, but real. A tool that has already fired eagerly can't be un-fired. We added a per-tool `idempotent` flag: non-idempotent operations — payments, destructive commands, outbound emails — fall back to the classic path and only fire after `message_stop`. The eager fast path is reserved for reads and safe operations. The second was **cancellation**. When the user interrupts or the model emits a stop sequence mid-stream, every in-flight eager tool needs to cancel cleanly. Leaking a tool execution into a dead conversation is both a correctness bug and a resource leak. We tied each dispatch to a cancellation scope whose lifetime matches the stream reader — stream dies, scope cancels, tools abort. Then a **flaky S3 list operation crashed an entire agent turn for a minute** before we realized we had no exception boundary around individual tool dispatches. One failing tool was bringing down the stream reader. Now every sealed dispatch runs in its own exception boundary; failure produces an error `tool_result` and nothing else propagates. Finally, eager execution **hides behavior inside the stream** in a way classic execution doesn't. "Why is tool X slow?" becomes unanswerable without knowing when it was sealed versus when it finished. Every seal event now emits an observability span with `seal_latency_ms`, `tool_call_id`, and `conversation_id`. When something misbehaves, the timeline tells you exactly which stage was slow. --- ## 7. The Numbers We benchmarked 16 workloads representative of real agent traffic, comparing sequential baseline, parallel tool calling, and eager tool calling. P50 wall clock from request-received to final assistant message. The numbers below come from the open-source benchmark in [eager-tools](https://github.com/cloudthinker-ai/eager-tools/blob/main/bench/results.md) — synthetic, deterministic, reproducible by anyone via `make bench`. | # | Workload | Tools | Sequential p50 | Parallel p50 | **Eager p50** | Speedup vs parallel | Speedup vs sequential | | --- | ------------------ | ----- | -------------- | ------------ | ------------- | ------------------- | --------------------- | | 1 | Analytics query | 3 | 4.9s | 3.5s | **2.9s** | 1.21× | 1.69× | | 2 | Search & retrieval | 4 | 5.7s | 4.2s | **3.5s** | 1.20× | 1.63× | | 3 | Customer support | 6 | 8.6s | 5.5s | **4.5s** | 1.22× | 1.91× | | 4 | Deploy preflight | 7 | 29.9s | 17.0s | **14.2s** | 1.20× | 2.11× | | 5 | Incident triage | 9 | 17.6s | 9.5s | **6.5s** | 1.46× | 2.71× | | 6 | Security sweep | 10 | 37.8s | 14.0s | **10.2s** | 1.37× | 3.71× | | 7 | Research synthesis | 10 | 22.5s | 10.0s | **8.0s** | 1.25× | 2.81× | | 8 | Lead enrichment | 7 | 13.4s | 7.0s | **5.5s** | 1.27× | 2.44× | | 9 | Content moderation | 8 | 19.5s | 13.0s | **10.0s** | 1.30× | 1.95× | | 10 | DB migration | 9 | 20.9s | 11.0s | **8.9s** | 1.24× | 2.35× | | 11 | Release notes | 10 | 19.6s | 9.5s | **6.7s** | 1.42× | 2.93× | | 12 | Legal review | 13 | 28.4s | 11.5s | **9.4s** | 1.22× | 3.02× | | 13 | Sales outreach | 14 | 23.6s | 11.0s | **8.6s** | 1.28× | 2.75× | | 14 | Ad campaign sweep | 15 | 30.4s | 11.5s | **8.8s** | 1.31× | 3.46× | | 15 | Invoice processing | 6 | 13.5s | 7.5s | **5.2s** | 1.44× | 2.60× | | 16 | Onboarding flow | 7 | 13.1s | 7.5s | **5.0s** | 1.50× | 2.62× | The synthetic harness removes network jitter so the comparison isolates the dispatch strategy — speedup vs parallel ranges from 1.20× to 1.50×, median ~1.28×. **In production, where tail latency and provider variance compound, end-to-end task completion is ~50% faster on the median trace, and longer tool chains pull further ahead.** Treat the OSS bench as the lower-bound version anyone can reproduce on a laptop; production wins are larger. ### Cost impact - **Output tokens per task:** effectively unchanged (same reasoning, same answer) - **Net cost reduction:** ~35% on tool-heavy workloads — fewer retries from timeouts, more conversations completing inside the cache TTL window, and faster end-to-end task completion freeing up agent slots. ### Where the speedup comes from Two stacked effects: 1. **Generation / execution overlap.** The biggest chunk. Every millisecond of stream time that used to sit idle on the tool side now has a tool running in parallel. 2. **Fewer turns.** A task that used to take three model turns often completes in one. Each saved turn saves prefill, network round-trip, and reasoning overhead. ![Bar chart of p50 wall clock for sequential, parallel and eager: 3-tool analytics query 4.9s, 3.5s, 2.9s (1.21× vs parallel); 9-tool incident triage 17.6s, 9.5s, 6.5s (1.46×); 15-tool ad campaign sweep 30.4s, 11.5s, 8.8s (1.31×)](https://engineering.cloudthinker.io/diagrams/agentic-system-design/bench-results-chart.svg "wide") --- ## 8. When NOT to Use It Being honest about the limits. - **Fast tools (sub-50ms).** If your tools finish in milliseconds, there is nothing to hide behind the stream. Seal/dispatch overhead exceeds the latency saved. Don't bother. - **Sequentially dependent tools.** If tool B needs tool A's result to even be formulated, the model won't emit B until A returns. No pipeline opportunity; eager and classic are identical. - **Non-streaming backends.** You can't seal per block without incremental parsing. If your provider or gateway buffers the full response before forwarding, eager tool calling is impossible until that changes. - **Non-idempotent tools.** Already covered above. Destructive operations, payments, outbound messages — these stay on the classic path. --- ## 9. A Mental Model That Helps Eager tool calling is **CPU instruction pipelining** for agents. The agent runtime is the CPU, the model stream is the instruction fetch, the tools are the execution units, and the seal event is the register-ready signal that lets execution begin before the rest of the batch is decoded. Once that analogy clicks, the rest of the design falls out of it naturally. --- ## 10. Try It Yourself ### Build it yourself Building this requires a streaming SSE parser, a seal detector, an async executor pool tied to the stream's lifetime, per-tool idempotency flags, and observability spans on every seal event. The open-source [eager-tools](https://github.com/cloudthinker-ai/eager-tools) library ships all of this. ### Closing Eager tool calling is not a novel idea in the abstract. CPUs have been pipelining instructions since the 1980s. What's novel is that the agent ecosystem spent two years treating streaming as a UX-only feature — something you do to make tokens appear live in a chat window — rather than as an opportunity to parallelize execution against generation. For any production agent running multiple tools per turn, this pattern is not optional. It is the difference between an impressive demo and a system fast enough to replace a human operator. *First published on [cloudthinker.io](https://cloudthinker.io/blogs/eager-tool-calling-50-percent-faster-agents) on April 15, 2026.* --- Previous: [CloudThinker Agentic Orchestration and Context Optimization](https://engineering.cloudthinker.io/journeys/agentic-system-design/01.md) Web version: https://engineering.cloudthinker.io/journeys/agentic-system-design/02/ --- # Generative UI Question: What should an agent emit when it needs to show someone a dashboard? Status: ongoing. Started 2026-04-06. Last entry 2026-04-06. 1 entries, 28 min read, 9.2k tokens. How our agents show results as dashboards and reports instead of walls of text, and what we learned moving that from generated images to a small UI language. ## What we think now Revision 1, 2026-04-06. 1. The agent describes what to show, it does not draw it. It picks from a closed set of 22 components and never emits a script. 2. The output format decides whether a dashboard can stream. JSON cannot be rendered until it is complete; a line-based language (OpenUI Lang) can be rendered line by line. 3. Moving from a Pydantic JSON schema to that language took a report from 30 to 40 seconds and about $0.50 to under 10 seconds and about $0.08, with 60 to 67% fewer tokens. 4. Store the language as plain text in Postgres. That gives diffs, replay and audit, and the same text renders to the web (React) and to PDF (HTML with WeasyPrint). ## Open questions - Can a dashboard be a shared, stateful artifact where a click works as a prompt? - Can the agent send a diff of the text instead of regenerating the whole dashboard? - How should interactive controls behave in a PDF or a chat message? ## Entries 1. [Generative UI in Production: Lessons from a Pydantic-to-DSL Migration](https://engineering.cloudthinker.io/journeys/generative-ui/01.md) (2026-04-06, start, 28 min, 9.2k tokens) Web version: https://engineering.cloudthinker.io/journeys/generative-ui/ --- # Generative UI in Production: Lessons from a Pydantic-to-DSL Migration Journey: [Generative UI](https://engineering.cloudthinker.io/journeys/generative-ui.md). Entry 1 of 1. Posted 2026-04-06 by Henry Bui. Type: start. _Our agent dashboards used to take 30-40 seconds and cost roughly $0.50 per report. Today they stream in under 10 seconds for around $0.08 — and they're built on the same design system as the rest of the product. This is the story of a migration we expected to take a week and ended up rethinking from first principles._ A year ago, asking a CloudThinker agent for a cost dashboard meant watching a spinner. The agent would call a `dashboard()` tool, the model would fill in a strict Pydantic schema field by field, and 30-something seconds later — once the entire JSON tree validated — a fully-formed dashboard would snap into place. It looked great. It used the platform's design system. It was also slow enough that most users would tab away and forget what they'd asked for. Today, the dashboard streams in. KPI cards render first with current-month totals. A line chart fills in as the agent emits it. A service breakdown appears as a bar chart. Everything still matches CloudThinker's design system. Total time: under 10 seconds. Total cost: about $0.08. The chart is interactive because it **is** the product, not a snapshot of it — click through to the expensive region and you drill in immediately. ![A cost dashboard streaming in over four frames: KPI cards first, then the full card grid, then charts rendering while the DSL is still arriving, then a recommendations table and gauges](https://engineering.cloudthinker.io/diagrams/generative-ui/artifact-streaming-storyboard.svg "wide") | Metric | Was | Now | Change | | --- | --- | --- | --- | | Wall-clock latency | 30–40s | ~10s | ~75% faster end-to-end | | Cost per report | ~$0.50 | ~$0.08 | ~84% cheaper per artifact | | Render mode | Atomic PNG | Streaming | Progressive, line by line | | Design consistency | Drifted | Locked | Enforced by construction | This post is about how we got from the first version to the second — and, more importantly, why the change wasn't a micro-optimization. It was a category shift the whole industry made at the same time, and it took us two architectures to land on the right one. --- ## Act 0 — The Two Detours We Skipped Before talking about what we built, it's worth being honest about what we _didn't_ build. When we first sat down with the "agent generates dashboards" problem, the two obvious paths were already on the whiteboard. We crossed both of them out fast. **Matplotlib PNGs.** The most popular pattern in 2023. Let the LLM write Python, run it in a sandbox, capture the PNG. Research from the [MatPlotAgent paper (arXiv 2402.11453)](https://arxiv.org/html/2402.11453v3) is worth quoting: it _"struggles to handle complex charts that require fine-grained numerical accuracy and precise visual-textual alignment of chart labels with plot components."_ A separate study, [_Are LLMs Ready for Visualization?_ (arXiv 2403.06158)](https://arxiv.org/html/2403.06158v1), documented that _"numerous attempts to modify the default configuration of scatterplots and bubble charts were unsuccessful."_ Five runs of the same prompt give five different results. There's a 4-8 second cold-start tax on `import matplotlib`. PNGs aren't interactive, can't be themed without a fight, and are effectively invisible to screen readers — the [ACM DIS 2025 GenUI Study](https://dl.acm.org/doi/full/10.1145/3715336.3735780) called accessibility _"a frequently-mentioned constraint"_ for generated output. And every chart means executing LLM-generated Python in a sandbox. The best way to not have a sandbox escape is to not execute the code in the first place. ![The matplotlib pipeline: user prompt, LLM writes matplotlib code, sandbox exec, matplotlib savefig, chat embeds PNG, with five failure modes alongside (design drift, stochastic output, 4-8s cold start, no interactivity, zero accessibility)](https://engineering.cloudthinker.io/diagrams/generative-ui/matplotlib-pipeline.svg) _The agent writes Python, a sandbox executes matplotlib, and a PNG embeds in chat. Five specific failure modes compound: design drift, stochastic output, cold-start overhead, zero interactivity, and accessibility gaps._ **Raw HTML / JSX freestyle.** _"Modern frontier models are great at React and Tailwind. Why don't we just let the agent write the UI directly?"_ Everyone has this idea. Hardik Pandya's essay [_Expose your design system to LLMs_](https://hvpandya.com/llm-design-systems) captures why it's a trap: LLMs _"fabricate token names, drift on values within a session, lose all context between sessions, and never notice when the upstream library ships breaking changes."_ Token economics collapse — a single dashboard becomes 2,000-4,000 tokens of markup before a single data point. Half-streamed HTML is broken HTML, so progressive rendering is impossible. Every dashboard becomes a one-off; the design system is a suggestion, not a contract. And then there's the security bomb. > **OWASP LLM05 — Improper Output Handling** > > Our injection vector wasn't hypothetical. Cloud resource names are user-controlled. A customer can tag an EC2 instance with an `onerror` attribute payload, and the next time our agent generates a dashboard referencing that resource, the tag flows through the LLM's output straight into `dangerouslySetInnerHTML`. > > The agent isn't even "attacked" — it's a pass-through for attacker-controlled strings in the most structurally dangerous way possible. The OWASP guidance is blunt: _treat LLM output as untrusted user input_. The only durable fix is to make it structurally impossible for the schema to represent a `