Cohen's Kappa

When two raters label the same items, some of their agreement happens by chance. Jacob Cohen introduced kappa in 1960 because percent agreement ignores that (McHugh, Interrater reliability: the kappa statistic). Kappa is (observed agreement − chance agreement) / (1 − chance agreement), so it is the share of the possible agreement beyond chance that the raters reached. It can range from −1 to +1; 0 means agreement no better than chance and 1 means perfect agreement.

The interpretation bands most often quoted come from Landis and Koch, who labeled 0.00 to 0.20 slight agreement, then fair, moderate and substantial, up to 0.81 to 1.00 almost perfect, and called the divisions “clearly arbitrary” (Landis and Koch, The Measurement of Observer Agreement for Categorical Data). McHugh argues that this kind of scale is too lenient for health research, since it implies a kappa as low as 0.41 might be acceptable, and recommends demanding more where wrong labels have real consequences. The same logic applies to AI evaluation data: decide the bar before you label, based on what a wrong label costs.

For AI teams kappa shows up in two places. First, any evaluation built on human labels is only as reliable as those labels; low kappa between annotators can indicate an ambiguous rubric or a subjective task, though it can also come from very imbalanced labels, so look at where raters disagree before interpreting any model score built on those labels. Two raters can also agree on the same wrong label, so high kappa is evidence of consistency, not proof of correctness (scikit-learn, cohen_kappa_score). Second, it is a direct way to check an LLM-as-a-judge setup: treat the model as one rater and a human as the other, and measure agreement on a labeled sample before trusting the judge at scale. LLM-as-a-judge vs human evaluation walks through that calibration. Cohen’s kappa is for two raters; McHugh lists Fleiss’ kappa as the adaptation for three or more.