Feature extraction

A model can’t read an email; it needs numbers. A classic spam filter extracts features such as how often each word appears, how many links there are and whether the sender is in the address book. A modern system might instead pass the text through a pretrained model and use the resulting embedding, hundreds of numbers that capture meaning. Either way, the downstream model trains on the extracted numbers, not on the raw text.

The big shift of deep learning was from hand-built features to learned ones, often called representation learning. In episode 63 at 24:08, Yutori’s Dhruv Batra describes that choice for web agents: rather than having engineers hand-design how to read a page’s underlying structure (“We’re not feature engineering,” he says), the model looks at the pixels and writes its own code to read the page. That describes his team’s design; it isn’t a general claim that hand-built features never help.

Extraction sets a ceiling on everything after it: if a signal isn’t captured in the extracted numbers, no later step can recover it. It can also be the most expensive part of a pipeline: running a large model to produce embeddings can cost more than training a small model on them. Feature selection is the different job of choosing among features you already have. Dimensionality reduction produces fewer features; it can create new combinations through extraction, as PCA does, or retain fewer original features through selection.