The first design decision in agent safety is where the enforcement lives. A rule stated in the prompt is a request; a rule enforced in code around the model is a control. Prompts get overridden by injection, drift under compaction, and fail probabilistically — so anything that must always hold (permissions, spend limits, destination allow-lists, transaction integrity) belongs in the deterministic layer the model cannot rewrite. The OWASP Top 10 for LLM applications reads as a catalog of what happens when this line is blurred: prompt injection, insecure output handling, and excessive agency are all failures of trusting the model layer with control-layer jobs.
The control stack, layer by layer
- Input/output validation. Schema-check tool arguments before execution; treat model output feeding downstream code as untrusted input that happens to parse.
- Per-call authorization. Every tool call is checked against this user, this task, this moment — not against what the model claims. Content must never grant itself capability.
- Least privilege, structurally. Read tools separated from write tools; scoped credentials per tool; the agent that ingests untrusted content doesn't hold the send-externally capability (the trifecta break from prompt-injection defense).
- Sandboxed execution. Code and file operations run resource-limited and network-restricted — an agent that can execute code can execute any code it was tricked into writing.
- Budget caps. Spend, iteration, and recursion limits turn a runaway loop from an incident into a log line.
- Approval gates on irreversibility. Sends, payments, deletions, deploys: a human confirms, so the worst reachable state without sign-off is a proposal.
This layering is what NIST's AI Risk Management Framework formalizes as govern/map/measure/manage — the useful discipline it adds is writing down, per capability, what the failure would cost and which layer catches it.
Classifiers are seatbelts, not brakes
Model-based safety classifiers (moderation endpoints, judge models screening outputs) add real defense in depth — they catch categories rules can't enumerate. But they are probabilistic, and a control that fails 1% of the time is not a control for permissions or money. Use classifiers to flag and filter; use code to forbid.
The test that matters
Red-team the deployed system, not the model: plant hostile instructions in the content your agent actually ingests (email, tickets, web pages) and verify the damaging action is structurally unreachable — not merely that the model usually declines.
Sources: OWASP — Top 10 for LLM Applications · NIST — AI Risk Management Framework.
Related: prompt injection defense, human approval gates, tool use, structured outputs.