In short: "Accepted without edit" rises when an AI system improves. It also rises when reviewers get tired, trust fluency, stop seeing hard cases, or ask for less. Read alone, it cannot tell these apart. A few cheap measures can.
Key points
- Fatigue, automation bias, selection of easy cases and narrowing ambition all push approval rates up.
- Experienced developers using AI were 19% slower yet believed they were faster, so self-report cannot rescue the metric.
- Blind re-review 3–5% of approved outputs and report the critical-error rate.
- Seed known-flawed items into the review queue each month and report the catch rate.
- A rising approval rate with flat or falling error detection is a warning, not a win.
It is the most popular metric in human-in-the-loop AI: the share of outputs approved without changes. It goes up over time, it is easy to compute, and it looks like learning. Sponsors like it.
Four other forces push it up just as reliably.
Fatigue. By the fortieth item of the afternoon, approval is a click. Vigilance is finite and it declines within a session.
Automation bias. As outputs become more fluent, people trust them more, whether or not they are more correct. Fluency is not accuracy, but it feels like it.
Selection. If routing sends the system only easy cases, its approval rate rises while its usefulness stays flat.
Narrowing ambition. Outputs drift toward safer, blander content because that is what gets approved without edits. There is less to correct, and less of value.
The perception problem
The randomised evidence is sobering. Experienced developers using AI on their own codebases were 19% slower than without it, and believed, before and after, that they had been faster. If experts misjudge their own speed, reviewers will misjudge their own vigilance. Self-report cannot rescue this metric.
Four measures that can
Independent error sampling. Take a random few percent of approved outputs and have a second reviewer assess them blind. The critical-error rate in that sample is the number that matters. Read approval rate only beside it.
Seeded errors. Insert a small number of outputs with known, material flaws into the review queue and measure how many reviewers catch. Quality-assurance and content-moderation teams have used planted test items for years. It is the most direct measure of whether the gate works.
Time on item. Look at the distribution of review time. Approvals in a few seconds on complex items are not reviews.
Ambition drift. Periodically compare outputs from month one and month six on similar cases for specificity: numbers, named risks, concrete recommendations. If later outputs say less, approval rates are rising for the wrong reason.
The rule
A rising approval rate with flat or falling error detection is a warning, not a win. Report the two together or not at all.
In practice
- Blind re-review 3–5% of approved outputs and report the critical-error rate.
- Seed a handful of known-flawed items into the queue each month and report the catch rate.
- Cap queue length per reviewer per session.
- Never put approval rate on a dashboard without the error rate beside it.
What would prove this wrong. Approval rates that track independently sampled error rates closely across systems, reviewers and time. Then approval rate is a sufficient proxy and the extra measures are overhead.
Common questions
Is approval rate a good metric for human-in-the-loop AI?
Not on its own. It rises when the system improves but also when reviewers tire, trust fluent output, see only easy cases, or accept blander output. Report it beside an independent error rate.
How do I check whether AI output reviewers are actually catching errors?
Insert a small number of outputs with known, material flaws into the review queue and measure how many reviewers catch. It is the most direct measure of whether the gate works.
What is ambition drift in AI outputs?
Outputs drift toward safer, blander content because that is what gets approved without edits. Compare month-one and month-six outputs on similar cases for numbers, named risks and concrete recommendations.
How much of approved AI output should be re-reviewed?
Blind re-review 3–5% of approved outputs by a second reviewer and report the critical-error rate.