escalation threshold calculator
Most AI workflows give each case a confidence score and send the low ones to a person. Where you draw that line is a cost question: set it too low and errors get through, too high and people review cases the model already got right. Put in how your cases spread across confidence bands, how accurate the model is in each, and what an error and a review cost. You get the cheapest threshold and every alternative next to it.
auto-complete at 0.95 confidence and above; send the other 45% to a person
| auto-complete | auto | reviewed | errors | review cost | error cost | total |
|---|---|---|---|---|---|---|
| nothing | 0% | 10,000 | 100 | $40,000 | $15,000 | $55,000 |
| 0.95 and above (cheapest) | 55% | 4,500 | 127.5 | $18,000 | $19,125 | $37,125 |
| 0.90 and above | 75% | 2,500 | 207.5 | $10,000 | $31,125 | $41,125 |
| 0.80 and above | 87% | 1,300 | 339.5 | $5,200 | $50,925 | $56,125 |
| 0.60 and above | 95% | 500 | 531.5 | $2,000 | $79,725 | $81,725 |
| everything | 100% | 0 | 751.5 | $0 | $112,725 | $112,725 |
| band | accuracy | auto, per case | review, per case | cheaper |
|---|---|---|---|---|
| 0.95 and above | 98.5% | $2.25 | $5.50 | auto |
| 0.90 to 0.95 | 95.0% | $7.50 | $5.50 | review |
| 0.80 to 0.90 | 88.0% | $18.00 | $5.50 | review |
| 0.60 to 0.80 | 75.0% | $37.50 | $5.50 | review |
| below 0.60 | 55.0% | $67.50 | $5.50 | review |
Break-even accuracy: 96.33%. A band the model gets right more often than that is cheaper to auto-complete than to review. A review costs $4.00 in time, and still lets through 1.0% of errors. Band accuracy is only as good as the sample behind it; check it with the accuracy proof calculator.
questions
how do i measure accuracy in each band?
Take a golden set: a few hundred recent cases a person has labelled carefully, including the hard ones. Run the model on them, group the cases by its confidence score, and count how often it was right in each group. Bands with few cases have wide error bars; the accuracy proof calculator shows how many cases you need.
why does calibration matter?
A threshold only works if higher confidence really means higher accuracy. Many models are overconfident: a score of 0.9 might be right 80% of the time. The calculator uses the accuracy you measured in each band, not the score itself, and warns you when accuracy does not fall as confidence falls. Re-measure after every model or prompt change, because calibration moves.
why isn't human review counted as error-free?
Because it isn't. Reviewers working through a queue miss errors, especially when the model is usually right and they stop looking closely. Measure it the same way as the model: give reviewers seeded cases with known answers. If review catches 99% of errors, 1% still get through, and that is part of what each review costs.
what if the law or a regulator requires a person to decide?
Then keep the person, whatever the numbers say. Credit, hiring, insurance, medical and other decisions with legal or similarly significant effects can require human involvement under GDPR Article 22 and human oversight under the EU AI Act. Use the calculator to size that review, not to remove it.
what should an error cost include?
What one undetected mistake costs once it reaches a customer or a ledger: the rework, refunds or credits, penalties, the support time, and a fair share of lost customers. If errors differ a lot in size, run the calculator once with the typical cost and once with the worst plausible one.
more free tools
- ai roi & token cost estimatorMonthly Claude API cost, cost per task and ROI for a workload
- eu ai act compliance auditorRisk tier, obligations and dates for one AI system
- agent workflow fit scorerIs this process a good fit for AI agents, and what shape?
- ai build vs buy calculatorThree-year cost: SaaS tool, custom agents, or keep it manual
- ai accuracy proof calculatorHow many test cases prove a vendor's accuracy claim?
- ai model routing calculatorWhat sending each request to the cheapest capable model saves
- human review staffing calculatorHow many reviewers an AI workflow's escalations need
- ai vendor scorecardWeighted comparison of AI vendors, with must-haves and a swing check
- gdpr dpia screener for aiDoes this AI use of personal data need a DPIA?
- agent loop cost calculatorWhat a tool-calling agent costs per task as its context grows
- llm judge agreement calculatorDoes your LLM judge agree with people beyond chance? Cohen's kappa
Want the numbers checked against your own process? Book a 20-minute call.