# Agentic system design

Question: How should we structure an agent system so it stays fast, cheap and reliable as tasks get longer and use more tools?

Status: ongoing. Started 2025-11-24. Last entry 2026-04-15. 2 entries, 32 min read, 11k tokens.

How we build the agent system itself: how agents hand work to each other, how prompts are laid out, and how tool calls are run. We measure latency, tokens and cost on every change.

## What we think now

Revision 2, 2026-04-15.

1. Start with one agent. Move to a flat supervisor with workers only when the numbers ask for it; a flat supervisor is enough up to about 10 agents.
2. Keep each agent's context separate and hand over short summaries (about 200 tokens), not whole transcripts. This cut tokens by 50 to 70%.
3. Prefer transfer over delegation when the worker can answer the user itself. It halved latency and cut one task from 10.2K to 7.2K tokens.
4. Lay out prompts for the cache. Static content first, then a checkpoint that moves down each turn, then the objective last. A cache hit rate under 90% is treated as an outage.
5. Compact the context at 70% full, by appending the summarize instruction instead of swapping the system prompt. That keeps the cache and was 80.7% cheaper ($0.3315 to $0.0639).
6. Start each tool as soon as its call has finished streaming, not when the whole message ends. About 50% faster on the median production trace, 1.20 to 1.50 times faster than plain parallel calls in the benchmark.
7. Do not start a tool early if it is not idempotent, takes under 50 ms, or needs another tool's result.

## Where we changed our minds

- 2026-04-15: ~~Run all tool calls in parallel once the model's message is complete.~~ Start each tool as soon as its own call is complete, while the model is still streaming. (entry 2: https://engineering.cloudthinker.io/journeys/agentic-system-design/02.md)

## Open questions

- Does hiding tool schemas the agent does not need hurt reliability?
- At what size does a flat supervisor stop being enough?
- Can a tool that is not idempotent be started early safely?

## Entries

1. [CloudThinker Agentic Orchestration and Context Optimization](https://engineering.cloudthinker.io/journeys/agentic-system-design/01.md) (2025-11-24, start, 20 min, 7.0k tokens)
2. [Eager Tool Calling: How We Cut Agent Latency by 50% on Long Tool Chains](https://engineering.cloudthinker.io/journeys/agentic-system-design/02.md) (2026-04-15, experiment, 12 min, 4.0k tokens)

Web version: https://engineering.cloudthinker.io/journeys/agentic-system-design/