Accuracy

Accuracy is the simplest score: correct predictions divided by total predictions. For a balanced, well-defined classification task it’s a reasonable headline number, and it’s what most people reach for first.

The trap is class imbalance. If 99% of cases are negative, a model that blindly says “negative” every time scores 99% accuracy and catches zero of the positives. That’s why accuracy alone is dangerous — it hides where the model actually fails. Pair it with precision and recall (or F1) to see the failures it conceals, and for AI agents, remember that a correct final answer says nothing about whether the agent took a sensible path to get there.

Same accuracy, different failures

Suppose 100 labeled tickets contain 10 urgent cases and 90 routine ones, and a positive prediction flags a ticket for review. One rule calls every ticket routine; another flags some:

Count	Always routine	Flagging rule
Correct flags (TP)	0	8
False flags (FP)	0	8
Missed urgent (FN)	10	2
Correct routine (TN)	90	82
Accuracy	90/100 = 90%	90/100 = 90%

Both make 10 errors. The first misses every urgent ticket; the second catches eight of them and adds eight unnecessary reviews. The headline number cannot tell those apart, so report the four counts and the class sizes with it, and decide how much a missed urgent ticket costs against a wasted review before choosing a rule.

Say how the score was computed, too. For multilabel data, scikit-learn’s accuracy_score computes subset accuracy: “the set of labels predicted for a sample must exactly match the corresponding set of labels” in the reference (scikit-learn, accuracy_score). That is stricter than counting each label decision separately, and the two can give very different numbers for the same predictions.