BERT and masked language models

Fill-in-the-blank exercises are the training game. Take “The bank approved the ___ application”, hide one word, and ask the model to guess “loan”. Because the model sees words to the left and the right of the gap, it learns representations that use the whole sentence. Devlin et al.’s 2018 paper applied this masked language modeling objective to a transformer encoder and called the result BERT, short for Bidirectional Encoder Representations from Transformers.

How it differs from chatbots. A chatbot is an autoregressive model that only looks left, so it can generate text one token after another. BERT is encoder-only: it reads and represents text well but does not write long answers. Teams usually fine-tune it with a small output layer for one job, such as sorting emails, tagging names or scoring search results. Many embedding models used for semantic search descend from BERT, and the BERTScore metric uses its representations.

On the show. In episode 59, at 8:05, Superhuman’s Loïc Houssier describes using large models to find useful email labels and then choosing to “fine-tune dedicated BERT models” for cheap inference at scale. That is his account of Superhuman’s approach. In episode 46, at 27:15, Galileo CEO Vikram Chatterji says his team first tried Google’s BERT for evaluation work and found that, compared with much larger language models, “the BERT model just doesn’t generalize.” Both episodes were sponsored by Galileo: the episode 59 sponsor read and episode 46 sponsor read name it, and Chatterji leads the sponsor.

Why it matters. Editorial synthesis from those two accounts: a fine-tuned BERT-sized model can be far cheaper and faster than a frontier LLM for a fixed, high-volume task, while a larger model may handle open-ended or shifting tasks better. That is the trade-off in generative vs. discriminative models and small language models.