llm judge agreement calculator
Before an LLM judge grades your agent, or an agent's own pass/fail call gates real work, check it against people. Label the same cases twice and count the four boxes. Raw agreement flatters a judge when most cases pass; Cohen's kappa takes out the agreement you would get by chance, and the interval tells you whether you labelled enough cases to trust it.
too few cases to tell
Kappa = (observed − chance) ÷ (1 − chance), with a large-sample interval; Landis and Koch bands. Raw agreement of 92.0% sounds high, but two raters who both pass 85% of cases would agree 74.5% of the time by luck. Watch the catch rate as well as kappa: a missed failure usually costs more than a false alarm.
questions
what is a good kappa?
On the Landis and Koch scale, 0.41–0.60 is moderate, 0.61–0.80 substantial and above 0.80 almost perfect. Use 0.6 as a floor for a judge that informs people, and aim for 0.8 before a judge gates work automatically. Check the lower end of the interval, not the point estimate.
why is kappa low when agreement is 95%?
If 95% of cases pass, two raters who say pass almost every time agree 90% of the time by luck. Kappa only credits agreement beyond that. This is the kappa paradox; the prevalence-adjusted figure (PABAK) is shown when failures are rare, and the catch rate tells you directly how many real failures the judge finds.
how many cases should i label?
Enough for the kappa interval to sit above your bar. The calculator shows how many cases give a range of about ±0.1 at your current agreement. A few hundred is typical, and the set must include the hard cases, or the judge is only tested on easy ones.
should the human labels come from one person?
Ideally two people label independently first. Their kappa with each other is the ceiling: a judge cannot be expected to agree with one person more than a second person does.
more free tools
- ai roi & token cost estimatorMonthly Claude API cost, cost per task and ROI for a workload
- eu ai act compliance auditorRisk tier, obligations and dates for one AI system
- agent workflow fit scorerIs this process a good fit for AI agents, and what shape?
- ai build vs buy calculatorThree-year cost: SaaS tool, custom agents, or keep it manual
- ai accuracy proof calculatorHow many test cases prove a vendor's accuracy claim?
- ai model routing calculatorWhat sending each request to the cheapest capable model saves
- human review staffing calculatorHow many reviewers an AI workflow's escalations need
- ai vendor scorecardWeighted comparison of AI vendors, with must-haves and a swing check
- gdpr dpia screener for aiDoes this AI use of personal data need a DPIA?
- agent loop cost calculatorWhat a tool-calling agent costs per task as its context grows
Want the numbers checked against your own process? Book a 20-minute call.