Active learning

Suppose an insurance company wants a model that flags suspicious claims. It has a million past claims but no time to have experts review them all. With active learning, the team labels a small starter set, trains a first model, and then lets the model point at the unlabeled claims it is least sure about. Experts label those, the model retrains, and the loop repeats. The claims the model already handles confidently never need a human.

How the model chooses. The survey by Burr Settles describes several query strategies. The most common is uncertainty sampling, where the model asks about examples whose predicted probabilities are closest to a coin flip. Others pick examples that a group of models disagree on, or examples expected to change the model the most. The person answering is called the “oracle”, usually a human annotator.

Where it fits. Active learning is a way to choose which examples to label for supervised learning: the learner chooses which examples to send for labeling, using a strategy such as picking the ones it is most uncertain about, instead of labeling a random sample. It complements approaches that avoid hand labels, such as weak supervision and synthetic data. It is one form of human-in-the-loop machine learning.

Why it matters. Expert data labeling is often the slowest and most expensive part of building a specialized model, especially in legal, medical or financial work. Active learning spends that budget where it helps most. Its limits, per Settles’ survey and editorial synthesis: the examples it picks depend on the current model, so early mistakes can bias what gets labeled, and the selected set may not be a fair sample for later evaluation. Keep a separately sampled test set.