AI red teaming looks for failures by deliberately challenging a system. Perez and colleagues studied using language models to generate test inputs that elicit harmful responses from another model (Red Teaming Language Models with Language Models). This can produce candidate attacks to inspect; it does not establish that an application is safe after the tested prompts stop working. Document the tested system, inputs and observed outcomes.
For a concrete example, test a document-reading agent with a retrieved page that asks it to email a secret to an unrelated address. Use synthetic data and controlled tools. Record whether it merely repeated the instruction, attempted a tool call or completed the action. Those are different failures, and their fixes may sit in retrieval, instruction handling or the tool’s authorization checks.
While discussing an algorithmic red-teaming process, Jeetu Patel says, “Can I actually jailbreak this model algorithmically” (episode 44, 12:35).
For builders, turn a verified failure into a regression case with a clear expected outcome. Preserve the conditions that made it possible instead of keeping only a cleaned-up final prompt. Prompt injection defines the input attack, evasion attack covers inputs crafted to slip past a classifier, and AI agent identity and permissions covers action boundaries. The evaluation hub connects adversarial cases with the rest of the testing process.
Sources
- Red Teaming Language Models with Language Models — Primary source for the explanation and distinctions in this entry.
