CloudThinker Agentic Orchestration and Context Optimization
1. Introduction
We wanted CloudThinker to do work, not just answer questions: notice a cost spike, fix the database config causing it, and send a report without anyone typing a word.
Single-agent designs hit a wall on long-running workflows. They cannot split specialized tasks, run operations in parallel, or control the token growth that comes with long reasoning. Left unchecked, an agent can spend $50 of tokens in ten minutes and still answer “I don’t know.”
With prompt caching, asynchronous context compaction and parallel tool calling, we got 80-95% lower cost, 7x faster task completion and 85% lower latency than our baseline. This entry describes the architecture and what we learned taking it to production.
2. Multi-Agent Architecture
2.1 Start with one agent
Always start with a single agent. Multi-agent systems are hard to debug. We moved to a team only when our single agent could not be an expert in Kubernetes, billing and security at once and started making mistakes.
We compared three approaches:
| Pattern | Coordination Model | Verdict |
|---|---|---|
| Single-Agent | No coordination needed | ❌ The Ceiling: good for simple tasks, fails on complex ones. |
| Network (Peer-to-Peer) | Distributed consensus | ❌ The Trap: agents argued in loops about who should check the database. |
| Supervisor | Centralized coordinator | ✅ The Solution: clear ownership. One boss, many workers. |
We standardized on the Supervisor Pattern, with Anna (General Manager) as supervisor: clear ownership, explicit routing, and one place for state.
2.2 Supervisor variant
| Variant | When to Use | Trade-offs |
|---|---|---|
| Flat Supervisor | Small teams (2-10) | ✅ Simple, reliable ← CloudThinker uses this |
| Supervisor (as tools) | Delegation | ⚠️ Less autonomy, tighter coupling |
| Hierarchical | Large orgs | ⚠️ Scales, but adds latency and middle management |
| Custom Graph | Complex dependencies | ❌ Flexible, but hard to debug |
Our domains (Compute, Database, Security) are distinct, so one flat layer is enough. A flat supervisor has the lowest coordination cost. Move to hierarchical only past about 10 agents, or when sub-teams must work fully on their own.
2.3 Optimistic routing with fail-safes
Routing every request through the supervisor is slow and expensive, so we assume the specialist can handle it and keep a safety net:
- Fast path: “Show running instances” goes straight to Alex (Cloud Engineer), with no orchestration step.
- Sticky sessions: if you are already talking to Tony about a database issue, your next message stays with Tony.
- Escalation: if Alex gets “Why is my RDS slow?”, Alex does not guess but escalates to Anna, who assigns it to the right specialist. Complex tasks get escalated instead of failing silently.
This cuts latency by 40-50% for most tasks.
2.4 Agent communication
Without rules, agents fill each other’s context with irrelevant data. Our group chat protocol enforces two:
- Explicit targeting: every message addresses someone (
@alex,@anna), or it is rejected. - Structured handoffs: summarize before handing over; no raw 50-page logs.
Work moves in two ways: delegation and transfer.
Delegation
In delegation the supervisor keeps accountability while workers do the specialized steps. It avoids two failures of naive chains: token growth when each agent inherits all earlier context, and lost task state at handoffs.
Example, an AWS cost investigation: Alex gathers EC2 spending data, Anna analyzes it and finds a database cost anomaly, and Tony investigates the RDS configuration. Tony needs Anna’s conclusion, not Alex’s CloudWatch output (50K+ tokens). So Tony receives Anna’s 200-token delegation message (“no compute correlation”), and Anna waits for Tony’s result to write the final plan.
Each agent keeps its own message history; only the handoff message crosses over:
- Agent A: 50K tokens → produces a 200-token summary
- Agent B: 200 tokens (A’s summary) + 30K (B’s work) = 30.2K tokens → produces a 200-token summary
- Agent C: 200 tokens (B’s summary) + 20K (C’s work) = 20.2K tokens
- Total: 100.4K tokens processed (56% reduction)
In production, context isolation gives 50-70% fewer tokens than shared context.
The cost is about 5 seconds of supervisor reasoning per handoff. The common mistake is delegating before the supervisor has framed the task: “@alex go check EC2 costs” without saying what to look for causes back-and-forth. When a worker fails mid-chain, the supervisor retries, picks another specialist or aborts; we retry 3 times with exponential backoff, then escalate to the user.
Transfer
In a transfer the supervisor hands off completely and exits; the user then talks to the specialist directly. Use it when the task needs one specialist and the supervisor will not need the result: “What’s our PostgreSQL version?” or “Show current Kubernetes pods.” Tasks like “Why did costs spike?” need delegation.
The supervisor’s state is archived, so there are no callbacks. Compared with delegation, transfer cuts tokens from 10.2K to 7.2K and latency by 50% (~4-6 s → ~2-3 s). The common failure is transferring too early:
- ❌
"App has high latency" → "@alex investigate"(too vague). - ✅
"API response times 200ms→1.8s at 14:00 UTC" → "@alex investigate infrastructure causes".
Transfer only when the specialist has enough context to work alone.
3. Context Optimization
More agents mean more tokens. We use four techniques: prompt caching, asynchronous context compaction, tool consolidation and parallel tool calling.
3.1 Prompt caching
A chatbot has about a 3:1 input-to-output token ratio; an agent can reach 100:1, because it plans and acts through many tool calls, often 50+ per task. Input tokens dominate cost.
Prompt caching lets the model reuse context it already processed. Cache writes cost a bit more, but cache reads are usually 80-90% cheaper and up to 85% faster, with a much shorter time to first token.
The rule: cache everything static, keep dynamic content at the end. “Static” is easy to break: as the Manus team found, one timestamp in the wrong place invalidates the whole cache. We track cache hit rate as a top-level metric and treat a drop below 90% as an outage.
Following Manus, we keep the objective prompt outside the cache boundary, in three tiers:
- Tier 1, static system prompt: tool schemas and core rules. Written once, then cache hits (95%+ hit rate across conversations).
- Tier 2, conversation history: actions and observations. The checkpoint moves forward after each completed turn, so all past work is cached.
- Tier 3, dynamic objective: always after the checkpoint. Live context (team, connected resources, active plan, retrieved memories), rebuilt every turn and never cached.
Only the small objective block is reprocessed each turn. Systems that edit the system prompt or insert content mid-history invalidate the cache on every call. This layout is the basis of the 80-95% cost reduction.
3.2 Asynchronous context compaction
Caching does not add space. A multi-hour infrastructure analysis can fill a 200K context window with tool output and reasoning.
We first summarized synchronously: at 90% full, the agent paused, a separate summarizer agent compressed the history, and the agent resumed. Two problems:
- Users waited about 45 seconds in the middle of a debugging session.
- Quality dropped. An outside summarizer lacked the working agent’s reasoning state and wrote generic summaries that lost technical detail; later tasks did worse in our benchmarks.
What we do now:
- Trigger at 70% in a background job.
- Don’t block: the agent keeps answering with the full context.
- Swap in: when the job finishes, the old history up to the trigger point is replaced by the summary; messages added since stay raw.
- Self-summarize: the active agent summarizes its own work (“Summarize your work so far”), so it keeps what matters for its current goal.
The pause is gone, and long tasks complete more often because recent context stays raw.
Don’t break the cache to summarize. The obvious approach is a new system prompt, “You are a summarizer.” That invalidates the cache, so you pay full price to reread 100K tokens. Instead, keep the system prompt and append the instruction as the last message.
Claude Sonnet 4.5 pricing: $3/MTok input, $15/MTok output, $0.30/MTok cache read.
❌ New system prompt, cache broken:
{
"system": [{"type": "text", "text": "You are a summarizer agent..."}], # ~500 tokens
"messages": [...] # 100K tokens, reprocessed as uncached input
}
500 tokens uncached ($0.0015) + 100K uncached ($0.300) + 2K output ($0.030) = $0.3315
✅ Same system prompt, instruction appended, cache kept:
{
"system": [
{"type": "text", "text": "You are Anna..."}, # ~8K tokens, already cached
{"cachePoint": {"type": "default"}},
],
"messages": [
..., # 100K tokens of history
{"cachePoint": {"type": "default"}},
{"role": "user", "content": "Summarize the conversation above. Extract: key decisions, "
"technical analysis, errors, current state."}, # ~500 tokens
],
}
8K cached ($0.0024) + 100K cached ($0.030) + 0.5K uncached ($0.0015) + 2K output ($0.030) = $0.0639, an 80.7% saving.
Summaries lose some nuance, so our template keeps technical decisions, error patterns and user feedback. In very long sessions, earlier summaries are included in later passes: older context gets more condensed while recent context stays detailed.
3.3 Tool consolidation
The Model Context Protocol (MCP) makes it easy to connect tools, and easy to end up with 500 of them. Their definitions crowd the context window, leave little room for reasoning, and lead to hallucinated parameters.
Anthropic’s code execution with MCP lets the agent discover tools from a file tree instead. It is efficient, but for us reliability comes first: we think an agent that does not see a “Restart Database” tool in its prompt is less likely to use it in a complex solution.
Our middle ground: consolidate the interface, hide the manual. Instead of five CRUD tools (recommendation_list, _get, _create, _update, _delete), there is one:
@tool(description=MINIMAL_RECOMMENDATION_TOOL_DESCRIPTION)
async def recommendation(
command: Literal["get_instruction", "get_all", "delete", "create", "update"],
recommendation_ids: list[UUID] | None = None,
recommendations: dict | None = None, # generic dict instead of a typed schema
) -> str:
The full schema sits behind command="get_instruction". The agent always sees that the tool exists, but pays for the parameter details only when it calls get_instruction. Documentation moves from always loaded to on demand.
3.4 Parallel tool calling
The ReAct pattern (2022) is sequential: call one tool, read the result, decide, call the next. When Claude 3 tool use became generally available in May 2024 it did not support parallel calls; current Claude models can call several tools in one response.
Models still default to sequential calls unless told otherwise. Anthropic’s documentation recommends: “For maximum efficiency, whenever you need to perform multiple independent operations, invoke all relevant tools simultaneously rather than sequentially.” That instruction alone can make multi-tool workflows 3-5x faster.
In CloudThinker this happens at plan steps. Finishing “gather cost data” can start “analyze compute patterns,” “review storage utilization” and “audit network traffic” at once. Each round trip also reprocesses the context, so batching 5 tool calls into one request removes 4 round trips. That cut costs by 30-40% on our complex workflows.
4. Conclusion
Multi-agent systems fail in predictable ways. The supervisor-worker pattern gave us predictable failures, traceable routing and measurable trade-offs.
Complexity is a choice. Start with a single agent, add prompt caching, and measure everything. Move to a supervisor only when the numbers call for it.
First published on cloudthinker.io on November 24, 2025.
Up: all entries | Next: Eager Tool Calling: How We Cut Agent Latency by 50% on Long Tool Chains >>