Many multi-agent systems start with the same intuition: if 1 model can answer a question, then several specialized agents should answer it better. Split the problem by source or capability. Let one agent search documents, another inspect repositories, and another pull from chat history, then merge the outputs into a final response. On simple tasks, that design works well enough to be convincing.
When user requests demand coordination instead of parallel lookup, adding agents doesn't produce better answers—it just multiplies conflicting contexts. Instead, they produce more partial answers, more duplicated work, conflicting findings, and more opportunities for the system to lose the thread. That was the architectural wall we hit, and it changed how we think about multi-agent systems.
Coordination, not the number or raw quality of individual agents, is the true scaling ceiling in multi-agent architectures. Flat architectures collapse as coordination requirements rise; moving from passive merging to hierarchical orchestration prevents that stall.
The flat pattern works until it doesn't
When we set out to build an enterprise troubleshooting assistant spanning documents, repositories, Slack, and infrastructure, our initial impulse was to give every data silo its own agent. The premise was straightforward: answer questions across multiple sources of team knowledge. To make this concrete, consider a request like "Give me information about my deployment." Behind that question, 3 agents have relevant knowledge. A Slack agent finds a thread from earlier that day where a team member reported an application wasn't responding. Another agent inspects the runtime cluster and confirms that an application container is stuck in a continuous restart loop following a recent deployment update. A repository agent traces that deployment update to a missing environment variable in the deployment manifest—a value that was present in staging but that the team never added to the production overlay. No single agent has the full picture. But together, they reconstruct a chain of events: a team member raised a concern, the infrastructure confirmed the failure, and the code revealed the root cause.
Questions like this expose the weakness of flat multi-agent systems: the answer lives in the relationships between sources, not in any individual source. Each agent retrieves correctly in isolation, but no one owns the reasoning across their outputs. That gap isn't a retrieval problem; it's a coordination problem.
Figure 1 illustrates this pattern: specialized agents fan out to their respective sources, and a final merger step combines the outputs.
It's the most intuitive multi-agent design, and the one we started with. But a single model (the merger agent) with a large prompt was never going to be enough. Different sources needed different retrieval paths, different tools, and often different prompting strategies. So the natural design was to introduce specialized agents.
For simple lookup queries, parallel retrieval worked well. The cracks appeared when users moved beyond simple multi-source lookups and began asking for reasoning across them. The system retrieved the right ingredients, but the final answer felt like several independent summaries stitched together. No component owned the task end to end. A merger can summarize outputs, but it can't decide which agent should investigate further, resolve contradictions, or trigger follow-up work.
The multi-agent orchestrator discovery
The fix wasn't better agents. It was replacing the merger with an orchestrator, as illustrated in Figure 2.
A merger receives outputs and combines them. An orchestrator plans before delegating, reviews partial results, asks follow-up questions, and reroutes work when something is missing. The failure in the flat design wasn't weak retrieval; it was the absence of anyone owning the task end to end. The orchestrator does.
The orchestrator also solves a problem users don't see: prompt quality. Users send vague requests. The orchestrator rewrites them into targeted, agent-specific prompts—a retrieval query for one agent, a Slack search for another, and a config audit for a third. Each agent gets instructions matched to its specialization, not the raw user input.
By refining prompts at the orchestration layer, the system improves every agent's output without requiring the user to know anything about what agents exist.
The result: the system can reason over dependencies between partial answers, notice when one agent's output raises a question for another, and request follow-up work instead of treating the 1st pass as final. That's the difference between simple retrieval and dynamic, multi-turn investigation.
The next scaling problem
The orchestrator solved coordination but introduced a new bottleneck: attention dilution. As the number of agents grows, it must evaluate more capability descriptions, make more routing decisions, and track more partial results. Rather than running out of context window, the orchestrator spreads its attention across too many choices. Every additional agent adds capability, but it also expands the orchestrator's decision space, degrading routing accuracy as prompt overhead grows.
That degradation shows up in recognizable ways. The orchestrator delegates too broadly. It delegates to the wrong specialist. It asks several agents to do nearly the same work. It underuses the most relevant expert because that agent is now one item in a long list of possible routes.
Flat orchestration scales poorly because each new agent adds another routing decision that the orchestrator must evaluate repeatedly, rather than simply adding capacity.
Why hierarchical orchestration helps
To break through the single-orchestrator bottleneck, we had to borrow a proven systems design principle: hierarchical orchestration.
Group agents into domain-level teams, each with its own orchestrator. A top-level orchestrator delegates to domain orchestrators, not to individual agents (Figure 3). This changes the system in 3 important ways.
First, it reduces the size of the decision surface at each layer. A top-level orchestrator no longer needs to choose from dozens of specialists. It only needs to choose between a handful of domains or subteams. That keeps routing focused.
Second, it preserves local context. A domain orchestrator knows the capabilities and tradeoffs inside its own area much better than a global orchestrator trying to manage everything at once. That makes delegation sharper and follow-up loops cheaper.
Third, it contains failure. If one branch of the hierarchy produces weak results, the problem is easier to isolate and correct than in a fully flat graph where all coordination paths compete in the same space.
Most importantly, hierarchy turns orchestration from a single overloaded decision point into a layered control structure. This layered hierarchy delivers practical scalability rather than mere organizational neatness.
Lessons learned
Adding agents is trivial; getting them to coordinate is where systems break. Three lessons stood out:
- Merging isn't coordinating. A system that collects outputs and stitches them together will plateau the moment questions require reasoning across sources. The shift from a passive merger to an active orchestrator was the single largest improvement we made.
- Orchestrators have limits too. A single orchestrator coordinating a growing set of agents will eventually spread its attention too thin. Routing quality degrades before the context window runs out. If the orchestrator can't reliably decide who should do what, more agents make the system worse, not better.
- Structure scales, agent count doesn't. Grouping agents into domain teams with their own orchestrators keeps decision surfaces small, preserves local expertise, and contains failure. Rather than acting as overhead, hierarchy provides the mechanism that allows the system to continue working as the agent set grows.
What's next
Hierarchical orchestration doesn't solve everything. It introduces additional LLM hops, which means trading round-trip latency for reasoning fidelity. For low-stakes aggregations, a flat merger remains cheaper, but when your system requires nested error recovery and cross-source root-cause analysis, the token cost of layered routing is far lower than the cost of failed runs and repetitive hallucinations. It also raises its own questions: how to evaluate routing quality, how to observe delegation failures, how to manage memory boundaries between layers, how to keep latency and token budgets under control, and how to recover when one branch of the system goes off course.
But the lesson is clear enough already: multi-agent systems don't scale by adding more agents. They scale by improving coordination. The 1st leap is moving from merging to orchestrating. The 2nd is realizing we must structure orchestration itself if the system is going to keep working as the agent set grows.
That is the pattern worth paying attention to. Not because it belongs to one product or one framework, but because it seems to be a general property of these systems. If we keep building multi-agent architectures as flat collections of specialists with a merger at the end, we will keep rediscovering the same wall.
