Skip to content

AI, decoded

What are AI agent guardrails, and how do you set them?

AI agent guardrails are checks and controls around requests, tool calls and responses. Some use a model to judge content; others validate arguments or enforce permissions. Set them by naming the failure you want to prevent, choosing the check and placing it before the effect it must stop. A model-based check can make mistakes, and a final-output filter cannot undo an action already taken. Keep action authorization outside the agent's control and test the paths, timing and failure behavior of the whole workflow.

· Updated · Chain of Thought

Level 4: How AI systems are built · 4.6 Cost, latency and safety basics

AI AgentsMulti-Agent Systems

  • 5 episodes
  • 5 guest clips
  • 7 glossary terms
A sticky note reading “don't do X” beside a barrier illustrates code-enforced action limits. The image says: “Don't” is a request. Guardrails live in code.

In detail

Separate a content judgment from an action permission

An agent can turn a mistaken judgment into a tool call that changes something. Guardrails address different parts of that path. Calling a check a guardrail tells you little about its mechanism or reliability.

NVIDIA’s NeMo Guardrails documentation includes checks that ask an LLM to assess content, and says their performance depends on the model. These checks can be useful while still missing an attack or rejecting a legitimate request. Code that invokes a classifier does not turn its judgment into a certainty.

Check or controlWhat it can decideWhat still needs testing
Model-based content checkWhether a request or response appears to violate a policyMissed violations, false alarms and behavior on new inputs
Argument validationWhether a tool call has the expected structure and valuesA valid call can still request an unauthorized action
Authorization or approval gateWhether a particular action is permitted before executionPolicy correctness, credential scope and coverage of every action path

Put decisive action checks before the effect

A response filter can withhold a final answer. It cannot rewind a tool action that already happened. Even an input check may finish too late if it runs alongside the agent. The OpenAI Agents SDK guide describes parallel input checking and the possibility of tool execution before cancellation, as well as checks before and after function-tool execution. Check the actual framework and tool path you use.

OWASP’s excessive-agency guidance recommends limiting capabilities and permissions, requiring approval for high-impact actions, and enforcing authorization in downstream systems. For an email-drafting assistant, that can mean exposing draft creation without send capability, or making each send wait for approval of the exact recipient and message. The credentials and every route to the send operation must respect that boundary.

The barrier in the illustration represents this kind of action limit. The broader guardrail toolkit also includes fallible content checks. Keeping enforcement outside the agent’s control is an architectural requirement to verify, not a guarantee conferred by a label.

Check the boundary

These fictional cases illustrate the design questions. They do not run an agent or establish that an implementation is secure. Choose an answer before opening each explanation.

A safety classifier approves an unsafe email, and the tool has send permission. Can the email still be sent?

Yes, in this example there is no other gate. The classifier's wrong approval does not remove the tool's authority. A recipient rule or an approval requirement would need enforcement before the send operation.

Every send checks valid approval for that exact message and recipient. Can a model's claim of approval satisfy the gate?

No, under this case's stated design the gate independently verifies approval. The model's claim is not that evidence. The implementation still needs tests for forged, missing and stale approvals, changed message details and alternate send paths.

An email has been sent when the final-output filter rejects the response. Does that rejection undo the send?

No. Withholding the response does not undo the completed send. Put the permission and approval checks before the action they must prevent; evaluate response checks separately.

Test coverage, timing and failure behavior

Our suggested checklist is to name the action and resource each check covers, then test a permitted case and a nearby prohibited one. Check tool calls, retries, handoffs and alternate paths. Decide what a timeout, unavailable checker or ambiguous verdict should do before releasing the workflow. For a required pre-action approval, execution must wait for a valid result.

Evaluate model-based checks on legitimate requests as well as attacks. OWASP’s prompt-injection guidance recommends layered mitigations and adversarial tests; it does not present a foolproof filter. The episode moments below provide the interview context for why these limits matter. Read agent identity and permissions for the surrounding authority model and sandboxing for execution containment.

Guardrails: judgment, validation and authorization Three illustrative controls for an email assistant make different decisions. A model-based content check judges whether content appears safe and can be wrong. Argument validation checks structure and values, not permission. An independent action gate verifies approval for the exact message and recipient before sending. A timeline places the required approval before the send and the response check after it. Rejecting the response does not undo an email already sent. Test the implementation, timing and every action path. Three guardrails, three different decisions Illustrative controls for an email assistant. Test their coverage and timing. CONTENT CHECKARGUMENT VALIDATIONACTION AUTHORIZATION Does this look safe?Is this a valid call?May this send proceed? A model can miss a violation.Evaluate attacks andlegitimate requests. A valid recipient fielddoes not prove sendingis authorized. Independently verify approvalfor this message and recipientbefore the send. EXAMPLE TIMELINE · REQUIRED APPROVAL MUST FINISH FIRST draftapproval gatesend emailresponse check A response check cannot undo an email already sent. Required action checks belong before the effect. Test credentials and alternate paths; a diagram or a label does not establish that an implementation is secure.
An illustrative comparison of checks, not proof of a secure implementation. Model judgments can be wrong; required action authorization must finish before the effect. Test coverage, credentials and alternate paths. Download the image

Hear it from the guest

“I feel like then in the next phase, a lot of companies kind of, like, took the LLM sometimes back and not let them have these interactions because they could just be talked into doing the wrong stuff too easily.”
Philipp Krenn, Elastic EP 32 · around 31:13 · read the transcript
“I think if we have the right guardrails in place, the LLMs will take a more front and center customer facing seat again, and if they can be trusted.”
Philipp Krenn, Elastic EP 32 · around 31:29 · read the transcript
“We're wanting to build highly predictable applications on top of a foundation, which is, by definition, unpredictable.”
Jeetu Patel, Cisco EP 44 · around 12:35 · read the transcript
“When we think about guardrails, it's typically like these prompts or these instructions that we can set in the system prompt for the model, but the model may choose to disregard those guardrails at some point”
Yash Sheth, Galileo EP 9 · around 14:00 · read the transcript
“What I see people trying to do is apply deterministic rules to inherently autonomous systems.”
Kris Lovejoy, Kyndryl EP 62 · around 11:58 · read the transcript
Watch on YouTube

Quotes lightly edited to remove filler words.

From the conversation

This explainer is drawn from these episodes — each carries its full transcript.

Concepts in this explainer

AI GuardrailsToxicitySandboxingAI AgentGenerative AILarge Language ModelNon-Human Identity

Keep exploring