Multi-Agent System Architecture: When One Agent Is Not Enough

Multi-agent systems are fashionable, and fashion is a bad reason to add complexity. Every extra agent adds calls, latency, handoffs to design and places for errors to hide. So the first architecture question is whether you need more than one agent at all.

Signs a single agent has outgrown its job

• The system prompt covers unrelated jobs. Instructions for intake, research, writing and checking all live in one prompt and the model mixes them up.
• The context window keeps filling. Tool output and history from early steps crowd out what later steps need.
• Some steps are cheap and some are expensive. Simple steps are running on a large model because the agent has to be large for its hardest step.
• Work could run in parallel. The agent processes twenty items one after another when they are independent.
• Permissions differ by step. The agent holds a send or write tool during steps that should only read.

If none of these apply, keep one agent and improve its context engineering.

Three architectures

Pipeline

Agents in sequence, each handing a structured result to the next. Predictable and cheap. Most business processes fit here.

Hub and specialists

A router or coordinator sends each request to one specialist agent with its own prompt and tools. Good for systems that handle distinct request types, such as a phone agent that answers sales, support and booking questions.

Planner and parallel workers

A planner splits the task into independent subtasks and spins up workers, each with a fresh context. Good for research and bulk processing. Expensive in tokens, so use it when the speed or depth is worth paying for.

The cost of splitting

Each agent boundary needs a schema, a check and logging. Each agent call adds latency. Coordination itself uses tokens. Anthropic reported that its multi-agent research system used several times more tokens than single-agent chat, which it accepted because the tasks were high value. Make the same trade-off consciously.

How to build one

1. Start with a single agent and log where it struggles.
2. Split along the struggle: the step with different context, model size or permissions.
3. Define the handoff schema before writing the new agent's prompt.
4. Write the orchestrator as code with step limits and spend caps.
5. Compare cost and quality against the single-agent version on the same test set.

The detailed method for drawing agent boundaries is in agentic AI system design, and the coordination patterns are in multi-agent orchestration.

Multi-agent system examples

• Research: a lead agent plans, sub-agents search different angles in parallel, the lead writes the report from their summaries.
• Customer service: a router hands each conversation to a billing, technical or sales specialist with its own tools.
• Content: research, writing and review agents in a pipeline, each with its own context.
• Software tasks: a planner breaks work into tasks, coding agents work on separate files, a reviewer checks the result.

Multi-agent system architecture diagram, in words

Draw the orchestrator as code at the top. Below it, each agent as a box with its model, job and tools. Arrows between agents carry named schemas, never full transcripts. Knowledge and memory stores sit to the side, connected only to the agents allowed to use them. Checks and approvals sit on the arrows. If your diagram has an arrow labelled "everything", that is where the cost and confusion will come from.

A cost comparison to make before you build

Estimate tokens per run for the single-agent version and the multi-agent version. The multi-agent version is often cheaper per call but has more calls, plus coordination. If it is not clearly better on quality, speed or safety, keep the single agent.

How I approach it

Most systems on my systems page are pipelines with two to four agents, not large swarms. That is deliberate. Fewer, smaller agents with clean handoffs are easier to run, cheaper, and easier to fix when something changes upstream.

How to build a multi-agent AI system from scratch

1. Build the single-agent version and a test set.
2. Read its logs and find where context or tools overload it.
3. Write the schema of the brief the new agent will receive.
4. Build the new agent with only that brief, its own short prompt and its own tools.
5. Write the orchestrator as code, with run IDs, stored step outputs and limits.
6. Run both versions on the test set and compare quality, cost and latency.
7. Keep the split only if it wins.

Building this way means every agent exists for a measured reason, which is the opposite of starting with a diagram of ten agents and hoping they cooperate.

Frequently asked questions

What is a multi-agent system?

A multi-agent system is an AI system where several agents, each with its own role, instructions, tools and context, cooperate on a task, passing results between them under the control of an orchestrator.

When should I use a multi-agent system?

When one agent's prompt covers unrelated jobs, its context keeps overflowing, steps need very different model sizes or permissions, or independent work could run in parallel. Otherwise a single well-designed agent is cheaper and easier to maintain.

What are common multi-agent architectures?

A pipeline of agents in sequence, a hub that routes requests to specialist agents, and a planner that splits work across parallel worker agents. Many systems combine them.

What is an example of a multi-agent system?

A research system where a lead agent plans the work, several sub-agents research different questions in parallel, and the lead agent combines their short summaries into a report.

Are multi-agent systems more expensive?

Usually in total tokens, because of more calls and coordination. Each call can be smaller and cheaper, so well-designed multi-agent systems can still cost less than one bloated agent, but this needs measuring.

How do I build a multi-agent system from scratch?

Start with one agent and a test set, find where it struggles, split that responsibility into a new agent with a defined brief, orchestrate with code, and keep the split only if it measurably improves quality, cost or speed.