Skip to main content

Multi-Agent Systems: When They Help and When They Fail

Published · 22 min read
AI engineer, author of 23 peer-reviewed papers in applied machine learning

Multi-agent systems: when they help. Helps: a lead agent splits work across three parallel workers whose results combine, when parts run in parallel or the task needs more context than one window. Hurts: four agents in a chain, when each step depends on the last, as in most coding tasks. Start with one agent, and compare at the same token budget.

A multi-agent system splits one task across several LLM agents, each with its own prompt, tools and context window. The published results point both ways. Anthropic's research system beat a single agent by 90.2% on its internal evaluation. In a Google study of 180 agent setups, every multi-agent design lost 39% to 70% against a single agent on a planning task.

The difference is the shape of the task, and how much extra compute the comparison allowed. This article collects what peer-reviewed papers and the engineering teams at Anthropic, OpenAI, Google and Cognition have reported. It covers the main orchestration patterns, accuracy against token cost, the failure modes that one study found in more than 1,600 annotated runs, and where the two protocols, MCP and A2A, fit.

We build AcruxCore, an open-source LLM-ops platform. The section on how AcruxCore handles this is ours; everything else is drawn from the cited research.

At a glance​

QuestionShort answer
What is a multi-agent system?Several LLM agents, each with its own instructions, tools and context, that work on one task and pass results or control between them.
When does it help?When the task splits into independent parts, such as broad research, or needs more context than one window holds.
When does it hurt?When each step depends on the step before, as in sequential planning and most coding.
What does it cost?Anthropic reports about 15 times the tokens of a chat. Several studies find that much of the accuracy gain disappears when a single agent gets the same token budget.
Why do these systems fail?One study of 1,600+ annotated runs groups 14 failure modes into three categories: system design, misalignment between agents, and weak verification.
Do I need MCP or A2A?MCP connects an agent to tools and data. A2A connects agents that run in separate services. Agents in one program call each other directly and need neither.

What is a multi-agent system?​

Anthropic's Building effective agents (December 2024) separates two kinds of LLM system. A workflow runs LLMs and tools "through predefined code paths." An agent lets the LLM "dynamically direct" its own process and tool use. A multi-agent system puts several agents, or a mix of agents and fixed steps, inside one application.

Three research frameworks made the idea popular. AutoGen (Wu et al., COLM 2024) builds applications from agents that talk to each other. MetaGPT (Hong et al., ICLR 2024) and ChatDev (Qian et al., ACL 2024) give agents software-team roles and run them through fixed phases such as design, coding and testing.

Agent orchestration: five common topologies​

Agent orchestration is the code that decides which agent runs next, what it receives, and when the run stops. Most systems use one of five topologies, or a mix of them.

Five agent orchestration topologies. Supervisor: a lead splits the task and combines the results. Hierarchical: supervisors of supervisors. Handoffs: the next agent takes over. Peer debate: agents revise each other. Shared blackboard: agents share one store.

TopologyHow control movesPublished example
Supervisor (orchestrator-worker)A lead agent splits the task, calls workers, and combines their resultsAnthropic's research system: a lead researcher starts 3 to 5 subagents in parallel
HierarchicalSupervisors manage other supervisors, or role agents work in fixed phases under one processMetaGPT and ChatDev: role agents organised like a software team
HandoffsOne agent passes the whole conversation to another agent, which takes overOpenAI Agents SDK: a triage agent hands off to a billing or support agent
Peer debateSeveral agents answer, read each other's answers, and revise over roundsDu et al.: three agents debate for two rounds
Shared blackboardAgents read from and write to one shared store, and a controller picks who acts nextMetaGPT's shared message pool; Han and Zhang's blackboard system

Supervisor, or "agents as tools"​

Anthropic's guide calls this pattern orchestrator-workers: a central LLM "dynamically breaks down tasks, delegates them to worker LLMs, and synthesizes their results." OpenAI's A practical guide to building agents calls it the manager pattern, where each specialist is exposed to the manager as a tool.

Anthropic's How we built our multi-agent research system (June 2025) is the most detailed public report on this topology. A lead agent saves its plan to memory, starts subagents that search in parallel, and decides whether more research is needed. Our supervisor multi-agent tutorial builds a small version of this pattern: one router call that picks a specialist, which then runs its own tools.

Handoffs​

OpenAI's guide calls this the decentralized pattern. In the Agents SDK handoff docs, a handoff is a tool that the model sees with a name such as transfer_to_refund_agent. When the model calls it, the run continues on the new agent. By default, that agent sees the entire previous conversation. An input_filter can remove or rewrite parts of the history first. The OpenAI Agents SDK triage tutorial shows what such a handoff looks like in a trace.

The difference from the supervisor matters in practice:

Supervisor (agent as tool)Handoff
Who talks to the user after the callThe supervisorThe new agent
What the specialist receivesOnly the input the supervisor writes for itThe whole conversation, unless filtered
Where results are combinedIn the supervisorNowhere; the last agent answers
Typical useResearch, analysis, tasks that need one final answerCustomer service routing, where each specialist owns its part of the dialogue

Peer debate​

In Du et al. (ICML 2024), several copies of a model answer the same question, read each other's answers, and revise over several rounds. Liang et al. (EMNLP 2024) add a judge agent that ends the debate. They found the debate works best when it stops adaptively, and that an LLM judge may not be fair when the debating agents are different LLMs.

Shared blackboard​

A blackboard is a shared store that every agent can read and write. MetaGPT uses a shared message pool where agents publish outputs and subscribe only to messages that fit their role. In Han and Zhang (2025, preprint), a control unit picks which agents act from what is on the board, until the board shows consensus. The authors report the best average score among the systems they compared, with fewer tokens.

What the evidence shows: accuracy vs token cost​

The headline gains are real, but most of them come with much more compute.

Where multi-agent systems won​

  • Broad research. Anthropic's system, with Claude Opus 4 as lead and Claude Sonnet 4 subagents, beat a single Claude Opus 4 agent by 90.2% on Anthropic's internal research evaluation.
  • Debate on reasoning tasks. In Du et al., three agents debating for two rounds raised GPT-3.5's accuracy on 100 grade-school math problems from 77.0% to 85.0%, and on arithmetic from 67.0% to 81.8%. The authors list the extra compute as the method's main limitation.
  • Decomposable work. In Google's Towards a science of scaling agent systems (Google Research blog, January 2026, based on the preprint by Kim et al.), a centralized multi-agent design improved a financial-reasoning benchmark by 80.9% over a single agent.

What those wins cost​

Anthropic reports that agents use about 4 times the tokens of a chat, and their multi-agent system about 15 times. On the BrowseComp benchmark, token usage alone explained 80% of the variance in performance. The number of tool calls and the model choice explained most of the rest. Anthropic concludes that multi-agent systems "require tasks where the value of the task is high enough to pay for the increased performance."

What happens at equal compute​

Several studies give the single agent the same budget, or a simple sampling method, and compare again.

StudyComparisonResult
Smit et al., ICML 2024Several debate protocols vs self-consistency and other ensembling, mostly on GPT-3.5, on seven datasetsDebate, in its original form, did not reliably beat self-consistency or Medprompt. One debate protocol won after its agreement level was tuned. Debate needed more API calls
Li et al., TMLR 2024Plain sampling and majority voting, with more and more agentsAccuracy rose with the number of agents, with no communication between them
Tran and Kiela, 2026, preprintA single agent vs five multi-agent designs, with reasoning tokens held equal, on multi-hop questionsThe single agent matched or beat every multi-agent design
Jwalapuram et al., 2026, preprintAutomatically generated multi-agent systems vs chain-of-thought with self-consistencyThe generated systems underperformed and cost up to 10 times more
Kim et al., 2026, preprint180 configurations: one single-agent and four multi-agent designs, three model families, four benchmarksLarge gains on decomposable tasks; 39% to 70% losses for every multi-agent variant on sequential planning

Two points follow. First, running the same model several times and voting already gives part of the benefit, with no orchestration at all. Second, task shape decides whether coordination helps. The Google study found gains where work could run in parallel, losses where each step depended on the last, and more overhead as the number of tools grew. Its predictive model picked the best architecture for 87% of held-out configurations.

Same budget, fair comparison: a multi-agent system with its token budget, against one agent sampled three times and combined by majority vote on the same token budget. Outside tasks that split cleanly, the gain often disappears.

Key point: In a Google study of 180 agent setups, every multi-agent design lost 39% to 70% against a single agent on sequential planning, while it gained on tasks that split into parallel parts. Task shape decides whether more agents help.

Tran and Kiela predict that multi-agent designs win when one agent's context is used badly, or when more total compute is spent. Anthropic's case fits the first condition: the task needs more context than one window holds.

Why multi-agent systems fail: the MAST taxonomy​

Why Do Multi-Agent LLM Systems Fail? (Cemri et al., NeurIPS 2025) is the largest study of how these systems break. The authors annotated more than 1,600 traces from 7 open-source frameworks, including ChatDev and AG2. Human annotators agreed with a Cohen's kappa of 0.88, which is high. The studied systems failed on 41% to 86.7% of their tasks.

The resulting taxonomy, called MAST, has 14 failure modes in three categories.

Why multi-agent systems fail, in three categories. System design: step repetition, unaware of when to stop, disobeys task spec. Inter-agent misalignment: reasoning-action mismatch, task derailment, fails to ask for clarification. Task verification: incorrect verification, no or incomplete verification, premature termination.

CategoryShare of failures (sum of its modes)Most common modes
System design issuesabout 44%Step repetition (15.7%), unaware of termination conditions (12.4%), disobey task specification (11.8%)
Inter-agent misalignmentabout 32%Reasoning-action mismatch (13.2%), task derailment (7.4%), fail to ask for clarification (6.8%)
Task verificationabout 24%Incorrect verification (9.1%), no or incomplete verification (8.2%), premature termination (6.2%)

The category totals are our sums of the per-mode shares in the paper. The most common failures are simple: an agent repeats itself, does not know when to stop, or ends the run with a check that is missing or wrong.

Do better prompts and topologies fix it?​

The authors tried two fixes on two systems: clearer role prompts, and a changed topology.

System and benchmarkBaselineImproved promptsNew topology
AG2 on GSM-Plus, GPT-484.75%89.75%85.50% (not significant)
ChatDev on ProgramDev (32 tasks)25.0%34.4%40.6%
ChatDev on HumanEval89.6%90.3%91.5%

Both fixes helped, yet the best ChatDev setup still failed most ProgramDev tasks. The authors conclude that these failures need deeper changes in system design.

Failures the engineering reports describe​

The labs report the same kinds of failure from production:

  • Too many agents. Anthropic's early versions started 50 subagents for simple queries. The fix was effort-scaling rules in the prompt: 1 agent for a simple fact, 2 to 4 for a comparison, more than 10 only for complex research.
  • Conflicting decisions. In Cognition's Don't Build Multi-Agents (June 2025), two subagents built parts of a game that did not fit together, because neither saw the other's choices.
  • Errors that spread. In the Google study, independent agents that never checked each other amplified errors up to 17.2 times. A central orchestrator reduced this to 4.4 times.

Handoffs and context: what each agent should see​

At each handoff, one question decides what the next agent can do: how much of the history does it need?

OptionWhat the next agent getsRisk
Full historyEvery message and tool result so farThe window fills, and the agent inherits earlier mistakes
Filtered historySelected parts, such as the user's messages onlyIt may lose a decision it needed
A written briefA short task description from the leadThe brief may be vague, and the agent repeats or misreads work
Compressed historyA summary of earlier actions and decisionsA wrong summary carries its error forward

OpenAI's handoffs pass the full history by default, and Cognition argues that agents should share full traces. Anthropic's lead agent writes a brief instead. Its early briefs were short, and in one case two of three subagents duplicated the same work. Anthropic now gives each subagent an objective, an output format, tool guidance and task boundaries.

MCP vs A2A: where each protocol fits​

The topologies above need no protocol when every agent runs in one program. A supervisor calls a worker as a function, and a handoff passes the message history to the next agent. Two open protocols cover the connections that leave the program.

MCP (Model Context Protocol)A2A (Agent2Agent)
ConnectsAn agent to tools, data and promptsOne agent to another agent
IntroducedAnthropic, November 2024Google, April 2025
Governance todayAgentic AI Foundation, under the Linux Foundation (December 2025)Agent2Agent project, under the Linux Foundation (June 2025)
Main conceptsServers that give an agent access to tools and dataAn Agent Card that describes what an agent can do, and tasks with a lifecycle whose outputs are artifacts

MCP is how an agent uses a tool or data source that runs as a separate server. A2A is how an agent asks an agent in another service to do a task, even one built by another vendor. Google's A2A announcement says A2A "complements" MCP, and A2A runs on existing standards: HTTP, server-sent events and JSON-RPC.

A first multi-agent system rarely needs A2A, because its agents usually live in one codebase and call each other directly.

The counterargument: build one agent first​

The labs that build multi-agent systems also advise against starting with one.

  • OpenAI: "Our general recommendation is to maximize a single agent's capabilities first." Split only when an agent fails to follow complicated instructions or keeps choosing the wrong tool.
  • Anthropic: find "the simplest solution possible, and only [increase] complexity when needed." Its research system report also says multi-agent systems fit poorly where agents must share context or depend on each other, which includes most coding tasks.
  • Cognition: a single-threaded agent keeps one continuous context, so it follows both of their principles. A separate compression model can extend it when the context gets too long, though they call that hard to get right.

The equal-budget studies support this default, and the extra structure adds failure modes that a single agent does not have, such as derailment between agents. The exceptions follow the task shape:

SituationBetter default
Parts of the task can run in parallel and do not depend on each otherSupervisor with parallel workers
The information needed is larger than one context windowSupervisor, with subagents that return short summaries
Different parts need different permissions, models or spend limitsSeparate agents, each with its own key and limits
A conversation must move to a specialist who owns it from then onHandoff
Each step depends on the result of the step beforeOne agent
You want higher accuracy on a short reasoning taskSampling and voting first; debate only if it beats that at equal cost

To pick a design, start at the top and stop at the first question you answer yes to.

One agent or several? Build one agent first and measure it. Does each step depend on the last? Yes: one agent. Can parts run in parallel? Yes: supervisor with parallel workers. Too much for one context window? Yes: supervisor, with subagents that return summaries. Must a specialist take over the conversation? Yes: handoff. Otherwise: one agent.

How to tell whether a multi-agent design is helping​

  1. Run an equal-budget baseline. Give one agent the same total tokens, or sample it several times and vote. If it matches, the extra agents are not the cause of the gain.
  2. Measure cost per solved task. Record tokens and cost for each agent, not only for the run. A system 15 times more expensive needs a gain that justifies it, and a budget per agent stops a runaway loop from spending it all.
  3. Read the failing traces with MAST categories. For each failed run, ask whether it was a design problem, a misalignment between agents, or a verification problem. The fix differs for each.
  4. Look for repeated steps and missing stops. Step repetition was the most common mode in MAST, and unawareness of termination was third. A trace with the same tool call ten times, or a run that hits the step limit, shows them.
  5. Check every handoff payload. Read exactly what the next agent received. This needs a trace that keeps every agent's calls in one run, with the turns of a conversation grouped into one session. A brief that leaves out a decision is the start of a misalignment failure.
  6. Use a different model as the judge. Liang et al. found LLM judges may be unfair when the agents are different LLMs, so do not let one of the agents grade its own team. How reliable LLM judges are covers the other biases to check before you trust a judge's score. For multi-turn runs, the judge also needs the earlier turns of the conversation.
  7. Repeat runs. The AG2 results above carry a spread of about ±1 to ±2 points over six runs, so a small gain from one run proves little.

How AcruxCore handles multi-agent systems​

AcruxCore is not an orchestration framework. It has no agent graph, no handoff or supervisor runtime, and no A2A support. You write the orchestration yourself, or use a framework such as the OpenAI Agents SDK, CrewAI or LangChain. AcruxCore records each run, controls what each agent may spend, and evaluates the results.

Problem from this articleWhat AcruxCore does today
Seeing which agent did whatOne trace per run, with nested spans. Span kinds include agent, llm, tool and chain. Gateway calls join a trace through the x-trace-id and x-parent-span-id headers
Cost per agentEvery span records its model, tokens and cost, so a trace shows what each agent's calls cost
Framework-built agentsAn OTLP endpoint reads OpenInference spans. The Python SDK's register(instrument=[...]) covers the OpenAI Agents SDK, CrewAI and LangChain
Multi-turn runsTraces that share a session_id are grouped into a session
Different models and limits per agentEach gateway call names its own model. Give each agent its own virtual key, with allowed models, requests and tokens per minute, and its own budget
Testing a change on real conversationsDatasets built from sessions keep the earlier turns, and the evaluation judge sees that history

Some gaps are worth stating plainly:

  • There is no dedicated handoff span kind. A handoff from the OpenAI Agents SDK appears as a tool span.
  • There is no separate "agent name" field. An agent is a span of kind agent, named after the agent.
  • Online evaluation rules score single llm spans. They do not score a whole multi-agent trace or session.
  • AcruxCore runs an MCP server, so an MCP client can read traces and run prompts. It is not an MCP client and does not speak A2A.

The CrewAI trip-planning tutorial shows a two-agent crew traced this way, with one agent span per agent and their model and tool calls nested inside.

A starting checklist​

  • Build a single agent with good tools first, and measure it
  • Split only when the task has independent parts, or needs more context than one window
  • Compare every multi-agent design against a single agent at the same token budget
  • Choose the topology from the task: supervisor for parallel work, handoff for routing a conversation
  • Write explicit stop conditions, and a limit on how many subagents can start
  • Decide what each agent receives at a handoff, and check it in the trace
  • Add a verification step before the final answer
  • Record tokens and cost per agent, and compare cost per solved task
  • Classify failed runs by MAST category before changing prompts

The open question​

Almost every published gain for multi-agent systems compares against a single agent that got fewer tokens. The few studies that equalise the budget find much smaller gains, or none, outside tasks that split cleanly. What is still missing is a shared way to report multi-agent results with cost: accuracy at a fixed token budget, against both a single agent and simple sampling. Until benchmarks report that by default, a claimed improvement may only show that the system spent more.

Sources​

Research papers

Engineering reports and specifications