In short: Token costs get the attention. The number that decides whether a judgment system pays for itself is human review time per decision that actually helped. Expect it to rise before it falls, put the crossover date in writing, and treat a miss as a decision.
Key points
- Unit prices for equivalent AI performance fell roughly 280-fold between late 2022 and late 2024, yet usage grew faster.
- In most deployments, skilled people's review and adjudication hours cost far more than the tokens.
- Reviewer minutes per useful decision counts all human time and divides only by decisions judged helpful at the time.
- A gated system follows a J-curve: review time rises early, which is where most programmes lose their sponsor.
- In lowob's own desk modelling, economics only work once review time per client falls under half the first month's.
AI cost conversations focus on inference. That is understandable: token bills are new, visible and growing. Unit prices for equivalent performance fell roughly 280-fold between late 2022 and late 2024, yet listed software companies reported margin pressure from token spend in 2026, because usage grew faster than prices fell. One cut its full-year operating profit forecast by more than a fifth, citing cloud costs that included AI tokens.
Those numbers matter. For a judgment system they are not the biggest cost.
The human cost dominates
Every record needs a reviewer. Every conflict needs an adjudicator. Every falsifier needs someone to check it on the date. Every seeded error needs someone to catch it. In most deployments the hours of skilled people spent on review and adjudication cost far more than the tokens.
If review time does not fall as the record grows, the system is a service with software attached, and it should be priced and staffed as one.
The metric
Reviewer minutes per useful decision = total human time spent on review, adjudication and falsifier checks, divided by the number of decisions where the decision-maker judged, at the time, that the system's preparation helped.
Three properties make it the right headline.
It includes all the human cost, not just the approval click.
It counts only useful decisions, so a system producing volume nobody values scores badly.
It cannot be gamed by fatigue. Tired reviewers approve faster, but they do not judge more decisions useful; if anything, fewer.
Expect a J-curve
A gated system adds review in its early weeks. Precedent does not yet exist, the schema is being tuned, reviewers are learning the tool. The metric will rise before it falls. That is normal, and it is also where most programmes lose their sponsor, because nobody said it would happen.
So say it, in writing, at the start. Measure the baseline: minutes per decision before the system. Predict the peak. Set a date by which the metric must cross back below the baseline. If it does not, treat the miss as a decision point: change the design, narrow the scope, or reprice the work as a service.
What this means for margins
In our own modelling of an operator-run desk, the economics only work once review time per client falls to under half of what the first month requires. Above that, margins are thin. Below it, they are healthy. The ratio will differ for every organisation. The existence of a threshold will not.
In practice
- Measure baseline minutes per decision before launch, by decision class.
- Log all human time: review, adjudication, falsifier checks, error sampling.
- Record "did the preparation help?" at the moment of decision.
- Publish the crossover date with the business case, and hold the review on that date.
What would prove this wrong. Judgment systems in which review time per useful decision stays flat or rises indefinitely while the organisation still captures clear net value. Then the metric is measuring the wrong thing.
Common questions
What is the biggest cost of running an AI decision-support system?
Human review. In most deployments the hours skilled people spend on review and adjudication cost far more than the tokens.
How do you measure the ROI of human review in AI systems?
Use reviewer minutes per useful decision: total human time on review, adjudication and falsifier checks, divided by decisions where the decision-maker judged at the time that the system's preparation helped.
Why does review time go up after launching an AI system?
In early weeks precedent does not exist, the schema is being tuned and reviewers are learning the tool, so the metric rises before it falls. Measure the baseline, predict the peak, and set a date by which it must cross back below the baseline.
What should happen if review time does not fall?
Treat the miss as a decision point: change the design, narrow the scope, or reprice the work as a service. If review time never falls as the record grows, the system is a service with software attached.