Black-Box Model

You can see what goes into a black box and what comes out, but not what happens inside. A small decision tree that approves loans is the opposite: you can read every rule. A large language model with billions of weights is a black box in practice. Its developers know the architecture and the training data, but no one can point to the part of the network that produced a particular sentence or explain why a tiny change to the prompt changed the answer.

There are two meanings. A model can be a black box because it is too complex for people to follow, like most neural networks, or because the vendor won’t disclose how it works, as with a closed commercial model behind an API. Both make it harder to find errors, check for bias and give people affected by a decision a real reason for it.

The responses differ. Explainability tools approximate why a black box made a decision, NIST’s four principles set out what a good explanation needs, and mechanistic interpretability research tries to open the box directly. Cynthia Rudin argues that for high-stakes decisions, teams should use an interpretable model instead when one performs as well, rather than explain a black box after the fact. For generative AI, where no simple model can do the job, the practical answer is usually outside-in: strong AI evaluation, logging and human review.

In episode 20 at 27:43, Smartling’s Olga Beregovaya says that even with a carefully written prompt and fine-tuning data, “the model spits out the least expected output,” and that “there’s quite a bit of black box,” more so as models grow. Galileo sponsored the episode. In episode 57 at 24:46, Snorkel AI’s Alex Ratner recalls seeing deep learning bring “increasingly powerful, but black box model architectures,” which led his team to bet that data would become the center of AI development. Galileo sponsored that episode.