Inside a Real AI Guardrail System (And Where It Breaks)

AI guardrails are supposed to provide defense in depth.

This system has six layers.

And it still fails in ways that are structural, not accidental.

We studied Claude Code's complete guardrail architecture -- from the behavioral instruction at its foundation to the blast-radius framework at its surface. Six layers. Each addresses a different threat vector. Together, they represent one of the more thoughtful attempts to constrain an agentic AI system in production.

But the problem is not missing controls. The problem is how those controls interact.

Defense in depth assumes independent failure modes. In this system, layers share state, share decision logic, and rely on the same model. When one layer fails, others are more likely to fail in the same direction. That is not depth. That is coupling.

This article walks through the stack, then breaks it.

Section 1: The Guardrail Stack

+-----------------------------------------------------------+
|  Layer 6: Actions Framework (reversibility/blast-radius)   |
+-----------------------------------------------------------+
|  Layer 5: Secret Scanning (client-side, pre-upload)        |
+-----------------------------------------------------------+
|  Layer 4: Bash Security (command analysis, sandbox)        |
+-----------------------------------------------------------+
|  Layer 3: Permission System (rules, classifier, modes)     |
+-----------------------------------------------------------+
|  Layer 2: Trust Dialog (workspace boundary, MCP approval)  |
+-----------------------------------------------------------+
|  Layer 1: Behavioral Guardrail (CYBER_RISK_INSTRUCTION)    |
+-----------------------------------------------------------+
|               Model (Claude)                               |
+-----------------------------------------------------------+

Each layer makes an assumption. Those assumptions are not independent.

Layer 1: The Behavioral Guardrail

Assumes: the model interprets intent correctly.

At the foundation sits a single paragraph -- the CYBER_RISK_INSTRUCTION. The entire dual-use security policy, verbatim:

IMPORTANT: Assist with authorized security testing, defensive security, CTF challenges, and educational contexts. Refuse requests for destructive techniques, DoS attacks, mass targeting, supply chain compromise, or detection evasion for malicious purposes. Dual-use security tools (C2 frameworks, credential testing, exploit development) require clear authorization context: pentesting engagements, CTF competitions, security research, or defensive use cases.

Owned by Anthropic's Safeguards team (David Forsythe, Kyla Guru). Requires team review before modification. The source warns changes "can have significant implications for how Claude handles penetration testing and CTF requests."

This is a behavioral instruction, not a technical control. It works when the model interprets it correctly. It fails silently when it does not.

Layer 2: The Trust Dialog

Assumes: trust is established consistently across all entry paths.

The showSetupScreens function in interactiveHelpers.tsx gates access through four sequential checks: workspace trust dialog, MCP server approval, CLAUDE.md external include warnings, and deferred environment variable application. The code comment at line 184: "Apply full environment variables after trust dialog is accepted... This includes potentially dangerous environment variables from untrusted sources."

The trust dialog is the front door. It is well-designed for the interactive case.

Layer 3: The Permission System

Assumes: policy is deterministic and complete.

Seven permission modes from default (ask for everything) to bypassPermissions (effectively root). Eight rule sources from policy through session. A two-stage AI classifier for auto-mode. Dangerous permission stripping. Denial tracking circuit breakers. Pattern-matched tool specifications with server-level MCP wildcards.

This is the most complex layer. See Article 4 for the full analysis.

Layer 4: Bash Security

Assumes: syntax-based detection is sufficient.

bashSecurity.ts blocks command substitution patterns: $(), ${}, <(), >(), =() (zsh), $[], ~[]. A dedicated ZSH_DANGEROUS_COMMANDS set blocks 20+ zsh-specific commands -- zmodload (gateway to dangerous modules), emulate (eval-equivalent), all zsh/system builtins. destructiveCommandWarning.ts flags git reset --hard, rm -rf, DROP TABLE, terraform destroy. filesystem.ts protects .gitconfig, .bashrc, .mcp.json, .git/, .claude/ with case-normalized path comparisons.

Layer 5: Secret Scanning

Assumes: secrets are detectable via regex.

secretScanner.ts performs client-side detection before team memory upload. Curated high-confidence gitleaks rules for AWS tokens, GCP keys, Azure secrets, GitHub PATs, Slack tokens, and more. The Anthropic API key pattern is assembled at runtime (['sk', 'ant', 'api'].join('-')) so the literal never appears in the bundle.

Layer 6: The Actions Framework

Assumes: the model reasons about risk correctly.

getActionsSection() generates behavioral instructions framing every action in terms of reversibility and blast radius. "A user approving an action once does NOT mean that they approve it in all contexts." Concrete examples of risky actions requiring confirmation. Closes with: "measure twice, cut once."

Section 2: Where It Breaks

These Are Not Bugs

These failure modes are not implementation errors. They are properties of the architecture. Each one emerges from the interaction between layers -- not from a missing check or a coding mistake.

Failure 1: Policy Precedence Confusion

Multiple sources of truth without a single authority.

Eight rule sources. The evaluation logic in getAllowRules, getDenyRules, and getAskRules flatmaps across PERMISSION_RULE_SOURCES without explicit precedence resolution between conflicting behaviors from different sources.

Deny is privileged over allow regardless of source -- generally the right security default. But a project-level deny rule can also block an admin-level allow rule, creating false-denies that are difficult to diagnose.

Concrete scenario: an enterprise admin deploys policySettings denying Bash(curl:*) to prevent exfiltration. A developer's projectSettings allows Bash(curl:*) for build scripts. The intended behavior: admin wins. The actual behavior depends on whether the deny rule is collected and evaluated before the allow rule in the flatmap chain. No single engineer can predict the outcome without tracing the full evaluation.

This is a classic security failure mode: multiple sources of truth without a single authority.

Failure 2: Classifier Uncertainty

A probabilistic model enforcing a binary boundary.

In auto mode, the classifier makes allow/deny decisions about tool use. No formal confidence threshold is documented. No fallback to human review when the classifier is uncertain. The system logs decisions for analytics but does not expose confidence to the user.

The denialTracking.ts circuit breaker detects when the classifier is behaving poorly in aggregate. It does not detect when a single decision is wrong. And it escalates to human prompting -- meaning the classifier's security decisions can be overridden by persistence.

The system uses a probabilistic model to enforce a binary boundary. That is not enforcement. That is approximation.

Failure 3: Extension-Driven Bypass

The permission system is not the final authority. The extension system is.

Hooks can produce permission decisions. The createPermissionRequestMessage function handles hook decision reasons. If a hook can block, it can also allow. A malicious hook overrides the permission system.

Plugin-provided MCP servers bypass the .mcp.json approval gate. A project .mcp.json plus a project .claude/settings.json with mcp__server1 in the allow list creates auto-approval for all server tools after one trust dialog click.

The compound bypass: a plugin installs hooks that influence permission decisions while providing MCP servers whose tools benefit from those decisions. The trust boundary is strong for MCP servers but weaker for hooks. The extension system operates outside the permission model (see Article 3 for the full attack taxonomy).

Failure 4: Trust Boundary Inconsistency

The trust boundary exists only in the interactive path.

The trust dialog sequencing in showSetupScreens is careful: workspace trust, MCP approval, CLAUDE.md warnings, environment variable application. But non-interactive sessions (-p flag for CI/CD) never reach showSetupScreens. The trust boundary is never established.

Outside the interactive path, the system assumes trust without re-establishing it. The bypassPermissions mode affects tool execution permissions but not workspace trust -- and if showSetupScreens never runs, the downstream MCP and CLAUDE.md checks may not execute.

The front door is strong. The side doors are not.

Failure 5: Complexity as Vulnerability

Each layer has internal complexity. Layer 3 alone has seven modes, eight sources, three behaviors, pattern matching, a classifier, dangerous permission detection, and denial tracking.

The interaction surface between layers is the real attack surface. A bash command must pass through Layer 6 (actions framework), Layer 3 (permissions), Layer 4 (bash security), and conditionally Layer 1 (dual-use tools). Each can allow, deny, or defer.

Complexity is not just hard to reason about. It is impossible to reason about exhaustively.

The PERMISSION_RULE_SOURCES constant: 8 sources. PermissionMode type: 7 modes. SAFE_YOLO_ALLOWLISTED_TOOLS: 20+ entries. COMMAND_SUBSTITUTION_PATTERNS: 12 entries. ZSH_DANGEROUS_COMMANDS: 20+ entries. DANGEROUS_FILES: 10 entries. DESTRUCTIVE_PATTERNS: 14 entries. The combinatorial space is vast.

And adding more layers does not help when the layers are coupled.

Failure 6: The Circular Guardrail

The entire dual-use security policy reduces to one instruction -- interpreted by the same model it is meant to constrain.

What constitutes "clear authorization context"? A user saying "I'm doing a pentest"? A CLAUDE.md stating "This is a security research project"? A directory named pentest-lab? The instruction cannot specify. The judgment is inherently contextual.

"Detection evasion for malicious purposes" requires the model to infer purpose. An attacker who frames their request as defensive research provides exactly the "authorization context" the instruction requires.

There is no technical enforcement. No classifier, no rule system, no sandbox. Just a behavioral instruction in the system prompt -- the exact channel that prompt injection attacks target.

The system relies on the model to enforce limits on the model. That is a circular dependency.

The Defense-in-Depth Illusion

Defense in depth assumes independent failure modes. In this system:

• Multiple layers rely on the same model (Layers 1 and 6 both depend on model judgment)
• Multiple layers share the same policy inputs (Layer 3 and hooks share permission context)
• Multiple layers evaluate the same action through shared code paths

When one layer fails, others are more likely to fail in the same direction.

That is not depth. That is coupling.

The guardrail system is not shallow. It is not carelessly designed. But coupled layers do not provide multiplicative security. They provide the illusion of it.

Section 3: The Lessons

Behavioral guardrails are not enforceable. They are advisory systems operating inside an adversarial channel. The CYBER_RISK_INSTRUCTION works because the model generally follows instructions. Not because there is a mechanism preventing violation. For any boundary that must hold under adversarial conditions, behavioral guardrails are insufficient as the sole control.

Policy complexity eliminates auditability. Eight sources, seven modes, three behaviors, pattern matching, classifier interaction, hook overrides. Can an administrator determine the outcome of a tool invocation for a given configuration? In principle, yes. In practice, the interaction of deny-before-allow, mode defaults, classifier decisions, dangerous permission stripping, and hook overrides makes the answer non-obvious for any non-trivial configuration.

The hardest problem is knowing when guardrails fail. A false-allow is invisible by definition. The action proceeds, no alarm fires, no log entry indicates an error. The system has no mechanism for detecting its own false-allows in real time.

What enterprises should demand:

1. Runtime monitoring -- real-time anomaly detection on permission patterns and tool usage
2. Policy simulation -- simulate a tool invocation against the full rule stack, see exactly which rules apply
3. Extension visibility -- complete inventory of hooks, MCP servers, and plugins with effective permissions
4. Independent layer validation -- adversarial testing of each layer in isolation and combination
5. Fail-closed defaults -- deny on uncertainty for security-critical decisions. The policy system currently fails open when policy fetch fails -- right for availability, wrong for security in high-risk environments (see src/services/policyLimits/index.ts).

Guardrails are supposed to make systems safer.

But in agentic AI, guardrails are becoming systems themselves -- complex, stateful, and difficult to reason about.

Right now, we are asking those systems to enforce security boundaries we cannot formally verify.

That is the real risk.

Not that guardrails fail. But that we cannot prove when they will.

Scott Thornton is an AI security researcher at perfecXion.ai, specializing in defensive research on LLM and agent vulnerabilities. All analysis was conducted on lawfully obtained, publicly distributed npm package code in an authorized research environment.