No human reviews this code, and the reviewer still stopped 46% of the PRs

This text is from the team document in one of my repos:

The system is fully autonomous: no human reviews code. Quality comes from gates that machines check.

I use this process on three private repos. Since 1 October, they have 731 merged PRs. That number shows that the agents do much work. It does not show that the work is correct. Thus, I counted the results of the gates.

The agents did the work, and I measured three repos

I took all the numbers from GitHub and from the agent session logs on 1 November 2026. Claude agents do the work. In repo A, all agents use Opus 5.5 since 10 October. Before that date, an agent could use an older model.

The agents now operate on one VM, on a Proxmox mini PC with an i7-14700T. I did not examine where the first PRs of repo A ran. The repos are private, thus you cannot examine my numbers. I use the labels A, B and C in place of the project names.

Repo	Type	Dates	Merged PRs	Median PR size (lines changed)
A	a dashboard product	1 to 29 Oct	476	368
B	the website of that product	12 to 14 Oct	124	488
C	a B2B platform with payment rules, not in production	26 Oct to 1 Nov	131	1,217

Look at the empty days. On 13 and 14 October, repo A has 0 merged PRs and repo B has 73. At that time I stopped the work on repo A and gave the machine to repo B. The chart does not show the cause of the other empty days.

One agent writes a PR. A different agent reviews it. Then a script, merge-gate, does the merge if two conditions are true:

1. CI is green.
2. The reviewer approved the latest commit.

The GitHub free plan cannot protect main on a private repo. Thus, this script is the only control.

The reviewer stopped 339 of 731 PRs

My first count was zero. The reviewers do not use the CHANGES_REQUESTED state of GitHub. This is their instruction, from the prompt template of repo A:

Post a formal review with gh pr review __PR__ --comment --body-file <file>; its first line
is exactly "REVIEW: APPROVED" or "REVIEW: CHANGES_REQUESTED", then numbered findings with
file:line and a fix.

Look at --comment. GitHub records each decision as a comment. Thus, I counted the first lines of the comments. A PR is “stopped” if it has one or more REVIEW: CHANGES_REQUESTED lines.

Repo	PRs stopped one or more times	Percentage	Maximum rounds on one PR	Median time, first stop to approval
A	187 of 476	39%	4	9 min
B	51 of 124	41%	2	12 min
C	101 of 131	77%	7	44 min
All	339 of 731	46%	7

I do not have a human baseline. Humans did not review these repos. I think that human teams stop fewer PRs, but I did not measure this.

At first, I thought that a high number was a good result. A retrospective changed my opinion.

One good sprint had 55 problems

The agents write a retrospective after each sprint. This line is from sprint 6 of repo A, in the section “Went well”:

- Reviews were short: 7 of 8 PRs passed in the first round.

After that sprint, an Opus 5.5 agent read the merged code and looked for problems. The retrospective calls this “the scan”. This text is from the same file, in the section “Didn’t go well”. I replaced a product name with [server] and removed the issue numbers:

- **Reviews missed a lot.** The Opus 5.5 scan after this sprint found 55 problems in the
  Sprint 5 and 6 code [...], three of them high severity. Some tests passed only because
  their fixtures used a shape the real [server] or SDK never sends [...].

Compare the two texts. A high pass rate can show good code. It can also show a weak review. The sprint data did not show which of the two was true. A separate scan of the merged code was necessary.

This high-severity bug is a good example. A health page had a “no traffic” state, and its test passed. The test fixture sent an empty array. The real server always sends 60 points, with requests: 0. Thus, a real user could not see that state.

There is one possible error in this comparison. The scan used the newest model, and some older reviews could use an older model. A better model can find more problems. I cannot separate these two causes.

A reviewer that breaks the code finds different bugs

Each agent reads a file with the rules for its job. The file for the reviewer agent changed after that sprint. I think that this new rule is the most important:

Break it: for each behavior the PR claims, remove or change that code and check that a test
fails, then put it back. A claim no test notices is a finding.
Run at least two mutations on the PR's main behavior, and list each one in the review with
what failed.

This is manual mutation testing. The reviewer changes the code and looks for a test that fails. In sprint 14, this procedure found tests with no value in 5 PRs. Two of these tests were for a tenant isolation check.

The sprint 19 retrospective gives this example:

commit on refusal: a settings change refused because its key was already taken still
committed the key's claim. The retry then answered 200 "saved" with nothing changed and no
audit entry.

The numbers do not show a clear improvement. The agents added the rule in steps, and the type of work changed during the month. These are the results for repo A:

Merge date	PRs stopped	Percentage
to 10 Oct	99 of 211	47%
11 to 17 Oct	23 of 97	24%
from 18 Oct	65 of 168	39%

The rule found real bugs. I do not know how many more bugs it found.

PR size explains much of the 77% in repo C

Repo C has payment rules. I thought that its reviewer was stricter than the reviewer of repo A. Then I sorted all 731 PRs by the number of lines changed:

PR size (lines changed)	PRs stopped	Percentage
less than 100	20 of 133	15%
100 to 499	78 of 238	33%
500 to 1,999	151 of 254	59%
2,000 or more	90 of 106	85%

The percentage increases at each step. Repo C has 56 of its 131 PRs in the last row. Repo A has 42 of its 476 PRs in that row.

Size does not explain all of the difference. With less than 100 lines, repo C stopped 9 of 19 PRs and repo A stopped 10 of 95. With 2,000 or more lines, the results are almost the same: 50 of 56 and 35 of 42.

Look at the two bars on the left. The difference between the repos is largest there, but repo C has only 19 PRs in that group.

The PR type also has an effect. These results are for repo A:

• feat: 110 of 165 stopped (67%)
• fix: 24 of 119 stopped (20%)
• refactor: 7 of 64 stopped (11%)

I suspect that PR type and PR size are related. I did not separate them.

I also tried to sort the review findings by keyword. This did not give a result. In repo A, 223 of 241 stop comments matched my “data integrity” keywords, and 230 matched “code standards”. The median stop comment has 481 words, thus it matches almost all keywords. A correct sort needs a person or an agent to read each comment.

The gate also failed because of the machine

The log of the merge gate in repo A has 176 results: 106 MERGED and 70 REFUSED. The log does not cover all of the project. One PR can have more than one result. Approximately half of the refusals name a check that failed. The other refusals are timeouts, crashes, or an exit code with no message.

A failed check does not always show a code problem. In sprint 6, a test file went over its time limit, but all its tests passed.

The retrospectives record these machine problems:

• The agents of a different project increased the load to between 64 and 106. One full gate had a timeout.
• /tmp/node-compile-cache increased to 4.3 GB. Three gates failed with EDQUOT.
• Usage limits stopped all agents two times in one sprint, for approximately 25 hours and 11 hours.
• Agents stopped while their background jobs continued to run. This occurred five or more times, and the agents did not read the results.

The main correction is a lock. This is from the gate runner script. I removed the project name from the paths and shortened the code:

flock /tmp/heavy-slots/slot1 flock /tmp/heavy-slots/slot2 \
  scripts/merge-gate "$pr" --update "$@" > "$S/gate-$pr.log" 2>&1

Look at the two flock commands. Agents run tsc and vitest through a wrapper that needs one of these two slots. A gate holds the two slots, thus the agents wait. This wait increased to approximately two hours when gates ran one after the other.

I did not measure if the lock decreased the failures. The lock is a queue on a small machine. It does not correct the cause.

I do not touch the code, but I control the product

I do not write this code, and I do not read the diffs. The team document gives a human only these tasks:

• Publish packages.
• Make a repo public.
• Buy items.
• Change accounts.

But I still send messages to the agents. I write in Turkish, so I searched the logs for Turkish text. The result is approximately 830 messages in the three repos. That is more than one message for each merged PR. This number is a maximum, because agents also write some Turkish text.

These are some of my messages, in translation:

• “The preview rail ring and the app colour do not match. Fix this too.” (with a screenshot)
• “They are OOMing. What is using the memory?”
• “Do you close the agents that completed their work?”
• “Is it done?”

These messages are not code reviews. But no gate checks a colour mismatch or idle agents.

All 731 PRs show my GitHub account. Thus, GitHub cannot show who did the work. For this section, my only evidence is the team document and my own process.

What I do not know

• The number of bugs that the reviews did not find. Repo A has 119 type:bug issues and 476 PRs. Of these issues, 65 name a merged PR. A name does not show that the PR caused the bug. Repo C has no bug label, thus I have no data for it.
• The result with real users. The README of repo C says Not production-ready. I did not examine real usage of repos A and B.
• Shared errors. The authors and the reviewers use models from the same vendor. I suspect that they have the same blind spots. I did not use a scanner from a different vendor.
• The cost. I did not calculate a cost for each merged PR. My token counts can be too high, so I do not give them here.
• The human result. No human reviewed these PRs, so I have no direct comparison with 46%.

My next test is a scan of the merged code in repo C with a model from a different vendor. I will compare its findings for each PR with the sprint 6 scan. If you use human reviewers as a control for agent review, please send me your stop rate for each PR size.