Safety starts with the effects of the system in its intended setting. NIST’s AI Risk Management Framework treats safety as a characteristic of trustworthy AI, including conditions in which systems should not endanger people, property or the environment (NIST, AI RMF 1.0). A model’s refusal behavior is one part of that work; the application’s tools and deployment conditions also need examination.
For example, a maintenance assistant could draft a repair plan for a technician to review. If you later let it execute commands on equipment, revisit the allowed actions, approval points and failure cases. Test what happens when the sensor data is missing or contradictory. Define a safe stopping state instead of rewarding the assistant for always completing a task.
In a discussion of how to divide responsibility between models and applications, Yash Sheth says, “have those guardrails at the application layer” (episode 1, 32:54).
That is his position on where to put controls, not evidence that one layer is sufficient. For builders, list the harms your workflow could cause, map each to a test or control and record the remaining limitations. AI red teaming covers deliberate stress testing, human in the loop covers review of consequential actions, and deploy AI agents safely covers moving from experiments into operation.
Sources
- NIST, AI RMF 1.0 — Primary source for the explanation and distinctions in this entry.
