Monday, November 16, 2026 · Week 47 DE · EN · FR · ES Dark
AI

How many coding agents can a review team handle?

Coding agents cleanly complete only every second task series in the BOBA-Bench. How teams demand proof of completion and manage review load.

By Alec Chizhik November 6, 2026 6 min read
How many coding agents can a review team handle?

In the new BOBA-Bench, coding agents solve on average 51 percent of 280 series of tasks, each of which builds on the previous one. A green test can pass even though the feature is faulty. For Addy Osmani of Anthropic, the number of agents should be guided by a team’s review capacity.

Key takeaways

  • Coding agents solve on average 51 percent of the task series in the BOBA-Bench. The measurement is based on prescribed tests in Python projects; each series comprises 20 consecutive tasks.
  • A green test can pass even though the feature is faulty. This applies to unit tests with test doubles when the test and the implementation share the same misunderstanding.
  • According to Osmani, the number of agents should be aligned with review capacity. He calls the gap between agent output and work that can actually be integrated the orchestration tax.

Related:Platform engineering is no longer a DevEx project

What the BOBA-Bench measures

The BOBA-Bench appeared on 1 November. The benchmark measures coding agents on tasks that build on one another. Each series comprises 20 consecutive tasks, and what gets assessed is the state of the repository after the last step. Ten coding agents took part.

On average they completed 51 percent of the 280 series successfully. For teams whose agent tasks build on one another, this is a timely finding. The advice that comes with the benchmark reads, in essence, “check first, then build on it”. The idea is to run the implementation or try out the feature before the next task.

The finding has limits. Only Python projects were measured. A result counts as correct if it passes the prescribed tests.

What is the orchestration tax? Addy Osmani uses the term orchestration tax for the structural gap between the output of coding agents and the work that can actually be integrated. His countermeasures include batched reviews and an agent count aligned with review capacity. Added to this is protected time in which people form their judgments on the agents’ work.

What a green check demonstrates

A green check shows that the existing tests pass. According to Addy Osmani’s analysis, a unit test built on test doubles can pass even though the feature is faulty. Test doubles are stand-ins that replace real dependencies such as databases or interfaces within a test. This happens when the test and the implementation share the same misunderstanding. Osmani is a Member of Technical Staff at Anthropic and works on Claude Code.

On the Claude Code team, agents run executable checks before a pull request, for example tests, builds or screenshot comparisons. They present the output of these checks.

At OpenAI, Codex receives a temporary observability stack for every isolated Git worktree in the internal development environment. A worktree is an additional working copy of the same repository. Observability in this context covers logs, metrics and traces, records of what happens in the running system. The agent queries logs with the LogQL query language and metrics with PromQL. Alongside test output, data from the running system can therefore also serve as evidence of completion.

Verification loops with tests and subagents

On the testing side, Osmani lists three methods:

  • End-to-end tests exercise real user flows through the entire system.
  • Property-based testing challenges invariants with generated inputs. The testing tool generates the inputs itself and checks whether the rule holds for every one of them.
  • When a system is replaced, the old and the new implementation run with identical inputs and the results are compared.

The second loop starts at review. On the Claude Code team, a subagent reviews the changes after the executable checks, launched with fresh context. A subagent is an agent started separately for a subtask. Fresh context means that it reads the change without the conversation history of how the change came about.

According to Anthropic’s own figures, the pull requests merged in the Claude Code repository more than doubled between June and September of this year. The share of pull requests subsequently reverted fell by more than 40 percent over the same period. Reverted here means that a change that had already been merged was undone again.

These figures come from an internal vendor report. Together with the BOBA finding, they argue for a close look at the respective verification method and the conditions under which a team uses it. Neither finding supports a blanket verdict on automated reviews.

Review capacity and system understanding

Osmani calls the structural gap between agent output and work that can actually be integrated the orchestration tax. His recommendations focus on the human side. The number of agents is to follow review capacity, reviews are batched, and protected time is reserved for human judgments.

A second term describes the understanding behind the code. Osmani defines comprehension debt as the growing gap between the code a system contains and the code people have actually understood.

To limit cognitive debt, three recommendations build on human understanding. Every AI-generated change that ships has been understood by a person. The reasons behind decisions are documented. The team regularly renews its shared understanding of the system through reviews or knowledge sharing.

Checklist for agentic coding teams

For everyday work with several agents, the practices described consolidate into seven checkpoints.

  • Verify before building on top. The implementation runs or the feature is tried out before the next task goes to the agent.
  • Executable evidence before the pull request. Test output, a build or a screenshot comparison is on hand.
  • Runtime data as evidence. Logs, metrics and traces from an isolated environment show how the change behaves.
  • Targeted test types. End-to-end tests cover the most important user flows, property-based tests the invariants, and comparison runs the system replacements.
  • A second look with fresh context. A subagent reviews the change without the history of how it came about.
  • Agent count follows review capacity. Reviews are batched and human judgments receive protected time.
  • Documented decisions. The reasons behind an AI-generated change are on record, and a person has understood the change.

Frequently asked questions

What does the BOBA-Bench not measure?

User intent, readability and production readiness lie outside the measurement scope. The generated task specifications have also not yet been through any human review.

Why do teams not rely on end-to-end tests alone?

In Osmani’s assessment, end-to-end tests are slower and more fragile than conventional unit tests. He therefore recommends a small selection of especially valuable tests.

Does working with agents weaken the ability to supervise them?

An internal Anthropic study from December 2025 notes that effective supervision of Claude requires coding skills. Anthropic describes the possible erosion of those skills through excessive AI use as the paradox of supervision.

Image source: AI-generated (November 2026)

Translated from the German original using artificial intelligence. The German version is authoritative.

Also available in

Deutsch
MBF Media Newsletter

The monthly briefing for decision-makers

Once a month, the MBF Media Newsletter gathers what matters from cloudmagazin, MyBusinessFuture, Digital Chiefs and SecurityToday, curated by the editorial team.

Around 23,500 IT and business decision-makers read this newsletter. Read along.

Subscribe for free
MBF Media Newsletter, aktuelle Ausgabe auf dem iPhone
A magazine by Evernine Media GmbH