Data Labeling

A hospital wants a model that spots fractures on X-rays. The scans alone are not enough; someone has to mark which ones show a fracture and where. That marking is data labeling, and the marked scans become training data plus a held-out set to test against. The labels are the ground truth the model is judged by.

Labeling powers supervised learning, where a model learns from input-and-answer pairs. It is also central to how chat models are tuned. In the InstructGPT work behind early ChatGPT-style models, labelers wrote example answers for fine-tuning and then ranked model outputs; those rankings trained the reward signal for RLHF. Today, labeling work ranges from quick tagging to expert reviewers grading code, legal analysis or agent behavior.

Why it matters: a model can only be as good as the examples it learns from and the answers it is scored against. Unclear instructions, rushed or inconsistent labelers and biased samples all flow straight into the model. Good programs write clear guidelines, have several people label the same items to measure agreement, and audit samples. Because labeling is slow and costly, teams also use automated or synthetic data approaches, which still need human checks.

On the show, Snorkel AI’s Alex Ratner says in episode 57 at 24:46 that labeling was long “viewed as janitorial,” and in the next turn at 28:05 estimates that 40 to 50% of the time behind his company’s data points goes into review and quality control, “which is basically labeling.” That is his account of Snorkel’s own process, and Snorkel sells data-development services. Episode 57 includes a Galileo sponsor read. In episode 63 at 27:45, Yutori’s Dhruv Batra recalls hiring human annotators to record themselves completing web tasks, and says that approach “does not scale,” which pushed his team toward collecting data with agents.