Jailbreaking

Jailbreaking targets a model’s safety training. The attacker isn’t hiding instructions in data the way indirect prompt injection does; they’re directly persuading the model to break its own rules — wrapping the forbidden request in a roleplay, a hypothetical, a translation task, or encoded text that slips past the filters. The goal is output the model is supposed to refuse.

It persists because safety is trained behavior, not a hard constraint, so there’s always another framing the training didn’t anticipate. Wei, Haghtalab and Steinhardt trace jailbreaks to two failure modes: the model’s capability goals compete with its safety goals, and safety training doesn’t generalize to every domain the model’s capabilities reach, such as encoded or translated requests. That’s why production systems don’t rely on the model’s refusals alone: they add independent guardrails — input and output filters that run outside the model — so a jailbreak that fools the model still has to get past a second check. Those checks reduce the risk rather than remove it: OWASP says it is unclear whether fool-proof prevention exists, which is why teams also test their systems with adversarial prompts. The two terms overlap: OWASP files jailbreaking under prompt injection, as input that makes the model disregard its safety protocols. The distinction that helps builders is the target. Jailbreaking subverts the model’s safety policy, while prompt injection in the narrower sense hijacks the application with instructions smuggled in through data.