Toxicity

A company that adds a chatbot to its customer forum does not want it insulting users, repeating slurs from the web or helping someone write a harassing message. Toxicity is the name for that class of output. Teams usually measure it with a classifier, a smaller model that reads a piece of text and returns a probability that people would find it toxic. The score can block or rewrite an answer at runtime, act as one of a product’s AI guardrails, or summarize how often a model misbehaves across a test set.

Researchers found the problem early in large language models. The RealToxicityPrompts study showed that ordinary-looking prompts taken from web text could lead pretrained models into toxic continuations, because the models had learned from web text that contained plenty of it. Filtering training data, RLHF and red teaming all aim to reduce it.

Toxicity scores are useful but blunt. Classifiers misread sarcasm, quoted abuse and reclaimed slang, and can flag text that mentions an identity group even when nothing harmful is said. What counts as toxic also depends on the product: a moderation tool has to read abusive text, while a children’s tutor should never produce it. A team should check a classifier against its own examples before trusting it.

In episode 37 at 4:12, Hamel Husain warns about vendors who pitch “a dashboard with all a bunch of generic metrics, hallucination score, toxicity score.” His view is that generic scores convince buyers more than they help, and that teams should start from error analysis of their own product’s failures. Galileo sponsored the episode, and Conor’s co-host for it is Galileo CTO Atindriyo Sanyal.