AI Security
Attacking AI systems, and defending them.
AI security covers the ways AI systems get attacked — prompt injection, data poisoning, jailbreaks, model extraction — and the defenses against them, from guardrails and red teaming to keeping autonomous agents inside safe bounds.
An agent's input can come from someone other than its user. Greshake and colleagues studied indirect prompt injection, where attackers place instructions in material an LLM application retrieves, and demonstrated attacks on integrated applications (Greshake et al., Not what you've signed up for). For a builder, an external document or tool result deserves a different trust boundary from an authorized instruction. Test what happens when retrieved material asks the agent to disclose data or take an unrelated action.
Charles Guillemet in episode 65 argues for hardware-enforced controls when agents can move money. Dan Lorenc in episode 68 discusses the pressure AI-assisted attacks put on patching. Can you secure an AI agent with software? develops the first conversation, while AI agent identity and permissions covers access boundaries. The prompt injection definition is a useful starting point for the input attack itself.
Start here
- Time to Exploit is Negative: AI Broke the Patch Cycle | Dan Lorenc, Chainguard
Hear how cheap models turn known vulnerabilities into exploits in minutes, shrinking the window for patches.
- You Can't Secure an AI Agent with Software | Charles Guillemet, Ledger
Examine hardware-enforced signatures for an agent policy engine.
Go deeper
6 episodes
- How Cisco Secures Personal AI Assistants for 90,000 Employees | DJ Sampath
- Time to Exploit is Negative: AI Broke the Patch Cycle | Dan Lorenc, Chainguard
- You Can't Secure an AI Agent with Software | Charles Guillemet, Ledger
- The First Fully Autonomous AI Attack is Coming | Kris Lovejoy, Kyndryl
- The Critical Infrastructure Behind the AI Boom | Cisco CPO Jeetu Patel
- Vercel's Playbook for AI Agents: From Vibe Check to Production | Malte Ubl
Explainers on this topic
- What Information Can You Share With an AI Assistant?
- Human in the Loop vs. Human on the Loop
- Agent Identity and Permissions: The Layer Nobody Built
- Can AI Agents Be Secured With Software Alone?
- Why Multi-Agent Systems Fail
- 4 Guardrails for Letting Your Whole Company Use AI
Terms on this topic
- Agent Swarm
- AI Gateway
- AI Guardrails
- AI Incident
- AI Red Teaming
- AI Safety Institute
- AI Supply Chain
- AI Watermarking
- Anomaly Detection
- Backdoor Attack
- Content Provenance (C2PA)
- Data Poisoning
- Deepfake
- Differential Privacy
- Digital Twin
- Evasion Attack
- Excessive Agency
- Facial Recognition
- Frontier Safety Framework
- Fuzzing
- Jailbreaking
- Least Privilege
- Mean Time to Remediate (MTTR)
- Membership Inference Attack
- Model Denial of Service
- Model Extraction Attack
- Model Inversion Attack
- Non-Human Identity
- PII Redaction
- Prompt Injection
- Quantum Computing
- Sandboxing
- Scheming
- Time to Exploit
- Token Leakage
- Toxicity
- Voice Cloning
- Vulnerability Chaining
- Zero-Day
Guests on this topic
DJ SampathDan LorencCharles GuillemetKris LovejoyJeetu PatelMalte Ubl