Single agent vs multi-agent: check the context before you split

After hybrid search and a reranker, my RAG system, GroundedDocs, still got [N] of 40 eval questions wrong or only partly right. The next step looked obvious: a planner agent, a retrieval agent, a verifier agent.

Before building any of it, I checked each failure for one thing: was the evidence in the model's context when it answered? For [X] of [N], it was not. The model had not failed to reason. It had never seen the text. More agents would not have fixed that.

TL;DR

Most "we need more agents" failures are context failures: the evidence, or the rule, never reached the model.
A second agent buys one thing, a separate context window. It does not make the model smarter, and Anthropic measured multi-agent runs at about 4x the tokens of a single agent.
Before you choose a shape, label your last 20 failed runs. Labels 01 to 03 are context problems. Fix retrieval, tool output, or history instead.
Pick the cheapest shape by asking three questions in order: steps known? Workflow. One context holds the job? Single agent. Pieces independent and read-only? Orchestrator with subagents.
Keep a split only if it beats one agent on a frozen question set against a pass mark you set before the run.

A second agent buys a context window, not intelligence.

The context is everything the model can read at the moment it answers.

A second agent gives you a separate context window. That buys isolation and the option of running work in parallel. It does not make the model more capable.

So the first question about a failed run is not "How many agents?" It is "What did the model see?" A model that never saw the evidence does not need colleagues. It needs the evidence.

Why the planner-retriever-verifier default fails

Five agent roles—planner, query, retrieval, verifier, and summarizer—work together around a whiteboard while an LLM asks whether anyone retrieved the actual evidence. Documents and an “actual evidence” note sit unused on the table, while a sign says “still wrong answer.”

Most agent tutorials answer hard failures with more roles. Production systems mostly do not look like that.

"Measuring Agents in Production" (Pan et al., 2025) surveyed 306 practitioners and ran 20 case studies. 68% of production agents run at most 10 steps before a human steps in. 70% use off-the-shelf models with prompting and no weight tuning. Teams name reliability as the top challenge and fix it with system design. Anthropic's own guidance says the same: find the simplest solution that works, which may mean no agentic system at all.

The common shape is an ordinary code that owns the control flow (step order, retries, budgets, and when to stop), with model calls where judgement is needed. The model's output is never deterministic. The path around it is.

Here is what a second agent costs:

Tokens. Anthropic reported agents at about 4x the tokens of a chat and its multi-agent system at about 15x. Against a single agent, that is close to 4x.
A lossy handoff. A subagent returns a summary, not what it read. Every decision it made along the way is invisible to the caller. This was Cognition's original case against multi-agent designs.
Compounding errors. A Google and MIT study of 180 agent configurations found coordination helps on parallel tasks and hurts on sequential ones. Independent agents working without talking to each other amplified errors 17.2x.
Harder debugging. One failed task becomes several traces you have to join. A Berkeley-led analysis of 1,600+ traces across seven multi-agent frameworks found 14 distinct failure modes.

If you have run backend systems, this is familiar. A subagent is a service boundary. You pay for serialisation (the summary), transport (the tokens), and distributed failure (the joined traces). The monolith rule applies: do not split until a measured limit forces you to.

How to diagnose before you split

Step 1: Pull your failed runs.

Take your last 20 failed runs, or your whole eval set if it is small. GroundedDocs uses a frozen set of 40 questions: 18 answerable, 11 partially answerable, and 11 it should refuse.

For each run, log two lists of chunk IDs: the ones that were actually in the prompt and the ones you labelled as the evidence.

Step 2: Check whether the evidence arrived.

This script labels each failure by that one question. It also prints recall on failed questions next to recall on the whole set, because the whole-set average hides exactly the cases you care about.

"""Label failed RAG runs by one question: was the evidence in the model's context?

Input: JSONL, one eval result per line, for example
{"id": "q07", "verdict": "fail", "context_ids": ["c12", "c40"], "gold_ids": ["c12", "c13"]}

verdict:     "pass", "partial" or "fail"
context_ids: chunk IDs that were actually in the prompt (after reranking and top-k)
gold_ids:    chunk IDs you labeled as the evidence; empty for questions it should refuse
"""
import json
import sys
from collections import Counter


def recall(row):
    gold = set(row["gold_ids"])
    return len(gold & set(row["context_ids"])) / len(gold)


def label(row):
    if not row["gold_ids"]:
        return "abstain: answered a question it should refuse"
    r = recall(row)
    if r == 0:
        return "01: evidence never arrived"
    if r < 1:
        return "01: evidence partly arrived"
    return "04: evidence present, answer still wrong"


def mean_recall(rows):
    return sum(map(recall, rows)) / len(rows) if rows else float("nan")


def main(path):
    with open(path) as f:
        rows = [json.loads(line) for line in f if line.strip()]
    failed = [r for r in rows if r["verdict"] != "pass"]

    for r in failed:
        print(f'{r["id"]:>6}  {label(r)}')

    print()
    for name, n in Counter(label(r) for r in failed).most_common():
        print(f"{n:>3}  {name}")

    answerable = [r for r in rows if r["gold_ids"]]
    failed_answerable = [r for r in failed if r["gold_ids"]]
    print(f"\nrecall, all answerable questions:    {mean_recall(answerable):.2f}")
    print(f"recall, failed answerable questions: {mean_recall(failed_answerable):.2f}")


if __name__ == "__main__":
    main(sys.argv[1])

Run it with python label_failures.py eval_results.jsonl. The full script is in the repo: [REPO LINK].

Step 3: Give every failure a label.

The first three labels are context problems. No number of agents fixes them.

01. The evidence never arrived. Confident half-right answers or refusals on questions the documents can answer. Fix retrieval: a second query for the part the first missed, a wider candidate set before reranking, or the parent section when a chunk boundary cuts the answer in half. Measure recall on failed questions only. This was my case.

02. Tool output floods the window. The agent is fine at step 5 and poor at step 20, and it repeats tool calls. Plot tokens per step across one long run; a steady climb means raw results are piling up. Store large results outside the context, hand the model an ID and a short summary, and let it load the slice it needs. Measure tokens per step and repeated tool calls.

03. The history is never compacted. The agent forgets a rule from the start or contradicts an earlier decision. Look for the rule at the step where it broke; it is usually still there, buried under thousands of tokens. Keep a short running record of decisions and open questions, and drop the raw transcript behind it. Measure rule violations per run.

The next two labels are the ones that do call for more.

04. The evidence was there, and the answer was still wrong. This needs more steps, not more agents: one agent that checks its answer and retries inside a hard step limit. Measure the pass rate on these questions and the cost per success.

05. Too much independent reading for one window. Dozens of sources, and the window overflows or the run is too slow in sequence. This is where a lead agent with parallel subagents earns its cost. Measure quality, p95 latency, and cost per success against a single agent.

If most of your runs carry labels 01 to 03, a second agent will fail the same way at a higher price.

Step 4: Pick the cheapest shape that works.

Four shapes, cheapest first. Each step up costs more tokens and is harder to debug.

Shape	step?	Use it when	Wrong when
Workflow	Your code	You can write the steps down in advance.	The path changes with every input.
Single agent	One model, in a loop with tools	Steps are unknown, and one context holds the job.	The path is known, so you pay for freedom you do not use.
Orchestrator + subagents	A lead agent; helpers return summaries.	Pieces are independent, read-only, and too big for one context.	Pieces depend on each other's decisions.
Peer swarm	Nobody owns the state.	You are experimenting.	You need to trace a failure or cap the cost.

Then ask three questions in order and stop at the first yes.

flowchart TD
    A[Failed run] --> B{Was the evidence in the model's context?}
    B -- No --> C[Context problem, labels 01-03:<br/>fix retrieval, tool output or history]
    B -- Yes --> D{Can you write the steps down in advance?}
    D -- Yes --> E[Workflow]
    D -- No --> F{Does one clean context hold the whole job?}
    F -- Yes --> G[Single agent with a hard step limit]
    F -- No --> H{Are the pieces independent and read-only?}
    H -- Yes --> I[Orchestrator with short-lived subagents]
    H -- No --> J[Stay with one agent and compact its history]

Three tasks, three answers:

Answering questions over your documents with citations is a workflow: retrieve, then answer. GroundedDocs runs this way by default.
Fixing a bug that touches several files is a single agent. The steps are unknown, but every change depends on the others.
Researching a broad question across dozens of sources is an orchestrator with subagents. This is where Anthropic reported a 90.2% gain over a single agent on its internal research eval, and the same team says work that shares context or depends heavily across agents is a poor fit.

Failure modes of this method

You check the candidates, not the prompt. The gold chunk can be retrieved and then dropped by the reranker or the top-k cut. If you log candidate IDs instead of the IDs that went into the prompt, a label 01 shows up as a 04, and you go build an agent for a retrieval bug.

Your gold labels are wrong. The whole diagnosis rests on the chunks you marked as evidence. A mislabelled chunk flips a run between 01 and 04. Hand-check five runs from each label before you trust the counts.

The label-04 retry loop has no brakes. "Check and retry" without a step cap is an agent that never stops. Put the step limit and the token budget in code, not in the prompt, and log every run that hits either one.

Three-panel illustration showing common agent evaluation failures: relevant evidence retrieved but missing from the final prompt, incorrect gold labels causing a misdiagnosis, and an unlimited retry loop continuing after repeated wrong answers.

What to keep, what to drop

The rule: before adding an agent, check what the first one saw.

To test a split before you keep it:

Run both versions on the same frozen question set.
Compare quality, p95 latency, and cost per successful task.
Set the pass mark before you run it.
If the split does not clear it, keep one agent and write down why.

"Never build multi-agents" is out of date too. Cognition published "Don't Build Multi-Agents" in June 2025 and in April 2026 wrote that it had shipped a few specific multi-agent flows. What ships is one orchestrator that owns the state and short-lived subagents that return a compressed result, not a group of peers negotiating?

What I am testing next

In GroundedDocs, deterministic RAG stays the default path. Only label-04 failures get a bounded agent path, built in LangGraph, with hard limits on steps and cost.

Inside that path I am testing one split: a single short-lived subagent doing an isolated retrieve-and-summarise pass, against the same work done by one agent. The subagent stays only if it clears the pass mark.

The cost and failure numbers above are published ones. Mine are next.

Next in this series: Keep the subagent only if it beats one agent: my GroundedDocs numbers. Quality, p95 latency, and cost per success for both paths. Subscribe to the newsletter to get it when it ships.