Mass Open, Agentic AI, Boston

Red Teaming Agents: The New Attack Surface of Tools and State

A practical threat model and mitigation pipeline for agentic AI systems

William Caban

Press F for full screen. S opens speaker notes. ? lists shortcuts.

Opening

One step at a time, nothing looks wrong

Your chatbot red-teaming checklist does not cover this.

Why this is a different problem

One prompt, one output vs. a loop with branches

Threat model walkthrough

Three new attack surfaces

Tool misuse

A benign-looking call to a real tool, chained toward an unintended effect.

Privilege escalation

Delegated actions accumulate access the agent was never meant to hold alone.

Memory / context poisoning

State carried across turns gets corrupted mid-task, not caught at the input gate.

Each step can look fine in isolation. Only the trajectory is the attack.

Why alignment alone doesn't close this

Alignment and guardrails solve different problems

	Model alignment	Guardrails
When it applies	Unanticipated situations needing judgment	Known failure modes: least privilege, isolation, policy
Scope	General, broad	Domain-specific, your own policy
Update cadence	Tied to model retraining	Independent, no retraining required
Enforceability	Best-effort, reduces risk	Auditable, deterministic, logged
Customization	None, baked into weights	Full, threshold tuning, domain rules

A well-aligned model still needs engineered, auditable controls for the three threat surfaces on the last slide. Neither tool substitutes for the other.

What actually works right now

Standalone attacker vs. hybrid planner + controller

Design principle for agent architecture

An agent should hold at most two of three powers

Untrusted input. Web pages, issues, dependencies, tool results.
Sensitive access. Private data, credentials, internal systems.
Outward action. State changes, messages, network calls.

Need all three in one session? Put a human approval or a validation step in the path. That step cannot depend on the model's own judgment.

Simon Willison named the combination the lethal trifecta (June 2025). Meta's Agents Rule of Two (October 2025) turns it into a design constraint.

Three powers: untrusted input, sensitive access, outward action. Any two are manageable. All three in one session is the dangerous combination: an attacker plants text, the agent reads it, and the agent has both the data and a way out.

Ask the room. Which of your agents hold all three? Most coding agents do by default. They read issues and repos, hold tokens, and run network commands.

The fix is to remove one power per session, or put in a gate that does not rely on the model. A research agent loses outward action. A browsing agent loses sensitive access.

If asked. 'Isn't a human gate just click fatigue?' It is if it fires often. Gate only rare, high-impact actions, and show the exact action and the data involved.

Source. Simon Willison named the lethal trifecta in June 2025. Meta published the Agents Rule of Two in October 2025.

Next. A structural pattern that enforces this.

Architecture pattern

Read with one model, act with another

Pattern from CaMeL (Google DeepMind, 2025). Untrusted text can change a plan's values, never the plan. Example policy: a value from an email cannot become the recipient of a send action.

From finding to demonstrated control

Four questions, each answered by one part of this pipeline:

What risks does this system have?
What guardrails are in place?
Are those guardrails working?
How do we keep improving?

Finding to engineering requirement

A finding is a spec for a detector

Each tool answers a different question, and all of their findings converge into one record. PyRIT: Microsoft, MIT, github.com/microsoft/PyRIT.

Three tools, three questions. garak probes the model endpoint, guardrails included. promptfoo red-teams the application layer. PyRIT is Microsoft's open-source framework for proactively identifying risks in generative AI systems. All three feed one record.

A finding is only useful once it becomes a spec for a detector. Where is it observable: at the input, mid-trajectory, or at the output? What would catch it there?

Point to. Signal, then detector. The unified record is where evidence lives. EvalHub is one open-source example of that record.

If asked. 'Why not one tool?' They see different layers and different attack styles, and teams usually already run more than one. The next slide maps the layers. It does not list PyRIT separately, so say it overlaps the garak and promptfoo rows.

Status. PyRIT is MIT-licensed and actively maintained. It had v1.1.0 in September 2026. It was not archived, whatever you may have read.

Next. Which tool for which layer.

Tooling

Match the tool to the layer under test

Layer under test	Tool	Question it answers	Typical output
Model endpoint, guardrails included	garak	Which known attack classes get through?	Per-probe pass or fail, attack success rate
Application: prompts, RAG, policies	promptfoo	Does the app hold under generated attacks and assertions on every change?	Test matrix with pass or fail per case
Models and agents on benchmarks	inspect-ai	How does it score on safety, security, and agentic tasks, inside a sandbox?	Logs, scores, transcripts per sample
Agent with real tools	MiDojo	Does the real agent resist payloads in the data and tools it reads?	Utility and security scores per task pair
Behavior under pressure	Petri	Does the model deceive, flatter, or help a harmful user across many turns?	Judge-scored transcripts for human review

garak: NVIDIA, Apache 2.0. promptfoo: MIT, joined OpenAI in March 2026 and stayed open source. Inspect: UK AI Security Institute, with a community catalog in inspect_evals. One tool is not a program. Each answers one question, and the record has to hold results from all of them.

What guardrails are in place?

Four types, rigid rules to flexible judgment

Pattern matching

Regex-based. Catches predictable data: SSNs, credit cards, account numbers. Instant redaction.

Algorithmic

Code-based logic: JSON schema validation, token limits, output structure checks.

Guardian models

Small classifiers alongside the main model. Catch jailbreaks, toxic content, prompt injections.

Vector-based

Converts input to embeddings, measures semantic distance from unsafe topics. Keeps a domain system on-topic.

Layer more than one type. No single type catches everything on its own.

Where guardrails get placed

Six checkpoints map onto the three threat surfaces

Tool checkpoint catches tool misuse · Infrastructure sandbox catches privilege escalation · Retrieval and Input catch context poisoning.

Who owns which control

Lower layers can narrow policy, never relax it

Organization baseline. Platform team. PII, injection, and language checks that are always on.
Application controls. App team. Topic limits and domain rules.
Agent and skill controls. Per-agent guardrails and sandbox policy.
Tool and action controls. Parameters checked before the call runs.

A lower layer can add checks or tighten limits. It cannot switch off what a higher layer set.

Same shape as IAM boundaries and network policy: an explicit deny higher up wins.

Guardrail design trade-offs

Auditability vs. coverage is a real trade-off

All four types from the last slide, placed by how easily you can audit them. LLM-as-judge is one way to run a guardian model. Programmable, rule-based policy (e.g. Colang / NeMo Guardrails) is one open-source example of the deterministic end.

Demonstrating remediation

Proving the control holds, not just claiming it

The trace loop back to Finding is itself evidence that feeds future findings.

Are those guardrails working?

Measure with escalating pressure, not one static test

Baseline: raw harmful prompt

+ System prompt override

+ Social engineering

+ Translation bypass

Adaptive LLM attack

Attack success rate = share of probes that bypass the guardrail. Lower is stronger. Each stage runs only if the prior one was blocked, so the ladder shows exactly where coverage actually ends.

Adaptive red teaming

Escalate when the defense holds, stop when it breaks

One classifier scores every response the same way, so strategies stay comparable. The last rung uses a challenger model that rewrites the prompt from the target's refusals. garak (atkgen, TAP-style probes) and promptfoo (iterative and tree strategies) implement this family.

Domain-aware red teaming

Turn your policies into your attack dataset

Example: a healthcare provider adds patient-data privacy, diagnostic misinformation, and off-label advice. None of them appear in a generic scan.

OWASP, NIST, and ATLAS tell you where to start. Your policies tell you what matters in your environment.

Evidence quality

Not every number is evidence

Kind	Examples	How to treat it
Direct measurement	Attack success rate, block rate, false positive rate, latency overhead	Computed from guardrail decisions and probe results. Safe to gate a release on.
Outcome proxy	Re-prompt rate, session completion, tool failure correlation	Advisory only. Correlation in traces is not proof of causation.

Version every metric definition and link it to its source signal.
Show data freshness next to the number.
Report uncovered probes as gaps. Never pass them by omission.
Mark payloads that never reached the agent as not applicable, not as a pass.

A defense that blocks attacks by making the agent useless is not a defense. Report utility next to security.

Promoting a control to production

Treat a guardrail package like a release

Thresholds are set per environment with a named owner. The package author cannot be the only approver. Adapting a definition creates a new package, which needs fresh evidence.

Where the harness meets the world

Three questions to ask about any agent harness:

What untrusted content does it read?
What can it run, and where does that code execute?
What credentials and networks can that code reach?

Agent harnesses

A harness runs untrusted text with your permissions

It reads issues, READMEs, dependencies, web pages, and tool output.
It runs shell commands, installs packages, and executes generated code.
It inherits your files, tokens, SSH keys, and network access.

Claude Code, Codex, OpenCode, OpenClaw, and Hermes Agent differ in the details. They share this profile.

May 2026: jqwik 1.10.0. The maintainer shipped a hidden instruction telling coding agents to delete the project's tests. ANSI escape codes hid it from the terminal. One agent flagged it and refused. A refusal is model behavior. It is not a boundary.

Source: Snyk and TechSpot reporting on jqwik 1.10.0, published May 25, 2026.

Where is the boundary?

Three ways to sandbox a harness

Credentials should never live inside the sandbox. Inject them at the network boundary so a hijacked agent has nothing to exfiltrate. The third mode is the cheapest to adopt and the weakest.

Three places to put a sandbox, with three different failure domains.

Left. The whole harness runs inside one boundary. Nothing it does escapes. This is the usual choice for laptops and CI.

Middle. The agent loop runs on a platform and each execution step gets a disposable sandbox. The agent can survive an execution failure, and execution never sees agent state.

Right. The loop runs with full access and only generated code is isolated. Many frameworks stop here. If the loop is hijacked, the host is exposed.

Credentials. Never inside the sandbox. A proxy can substitute real values on outbound requests so the sandbox holds only placeholders. Claude Code's sandbox proxy supports credential masking like this, and OpenShell injects keys at the boundary.

Next. How real harnesses compare.

Harness isolation

Permissions are policy. A sandbox is a boundary.

Harness	Default posture	What enforces it	Check this
Claude Code	Permission prompts. Optional OS sandbox for Bash and its child processes.	Seatbelt on macOS, bubblewrap on Linux. File limits plus a proxy with a domain allowlist.	Is the unsandboxed retry disabled? What can MCP servers and hooks reach?
Codex	Sandbox mode plus approval policy: read-only, workspace-write, or danger-full-access.	Seatbelt on macOS, Landlock and seccomp on Linux. .git stays read-only.	Which mode do scripts and CI use?
OpenCode	allow, ask, or deny rules per tool. Most default to allow.	Policy layer. The docs describe no sandbox, so add a container or VM.	Can bash read what a deny rule blocks?
OpenClaw	Docker sandbox for tools. Modes: off, non-main, all.	Container per session, agent, or shared. Separate tool policy and an elevated escape hatch.	Which sessions are sandboxed? Who can use elevated?
Hermes Agent	Approval for dangerous commands. Container backends.	Docker, Singularity, Modal. Approvals are skipped in containers because the container is the boundary.	Which backend runs in production? Are MCP credentials filtered?

Defaults change fast. Read each project's current documentation, then test the boundary itself, not the config file.

The column that matters is 'what enforces it'. A prompt asks the operator. An OS sandbox or a container stops the process whatever the model wants.

Walk down. Claude Code has an optional OS sandbox for Bash. An escape hatch retries a failed command outside it, goes back through the permission flow, and can be disabled. Codex's most permissive mode removes enforcement entirely, so watch for it in scripts and CI. OpenCode's rules are a policy layer, most default to allow, and its docs describe no sandbox. OpenClaw separates sandbox, tool policy, and an elevated escape hatch. Hermes skips approvals inside container backends on purpose, which is only safe if the container really is a boundary.

Careful. Do not say any of these is insecure. Say which layer enforces what, and what to verify. Defaults change fast. Re-check each project's docs before you present. This table was verified in September 2026.

Next. Layering the boundary.

Layering the boundary

Application and kernel isolation stop different failures

Attack	Kernel only	App sandbox only	Both
Prompt-injection exfiltration	Leaked	Blocked	Blocked
Kernel exploit, container escape	Blocked	Host compromised	Blocked

The application layer decides where data can go.
The kernel layer decides what a compromised process can reach.
Red-team and evaluation runners execute hostile content too. Give them the same stack.

Source: published test of OpenCode with an open-weight model on Kubernetes, using OpenShell and Kata Containers (Red Hat Developer, July 23, 2026). One agent, one test. Treat it as a pattern, not a benchmark.

Two threat classes do not overlap. Prompt injection turns into exfiltration over an ordinary network call, which a microVM does not inspect. A kernel exploit bypasses user-space policy entirely, which an egress proxy does not stop.

Point to. The table. In the published test each single layer lost to the attack in its blind spot. Only the combination stopped both.

It is one agent and one test. Treat it as a pattern, not a benchmark.

When is container isolation enough. A narrow service account, known endpoints, human approval for novel actions, and no writes to systems of record. Move to kernel isolation when the agent runs arbitrary code, holds broad credentials, or acts on sensitive data. Kernel isolation still needs correct network and resource limits.

Evaluation runs too. Red-team and evaluation runners execute hostile content as well. Give them the same stack. Inspect's default compose config sets network_mode none, which is the right instinct.

Source. Red Hat Developer, July 23, 2026: OpenCode with an open-weight model on Kubernetes, using OpenShell and Kata Containers.

Next. Now test the boundary between the harness and its tools.

Adversary in the middle

Put a proxy between the harness and its tools

The agent does not change and does not know. You test the real harness and model, not a copy. Open-source examples: MiDojo (MCP and other agent protocols) and AgentDojo (rebuilds the environment inside the framework).

Replace the real tool with a stand-in that sits in the path. For each call it can forward to the real tool, splice a payload into the result, change the tool list or descriptions, or record the call without running its side effects.

Then. The grader scores two things independently: did the agent finish its task, and did it resist the attack. Then it checks reachability. If the payload never arrived, the run is not applicable.

The advantage over copy-based benchmarks is that you test the actual harness, model, prompts, and sandbox you ship. The agent does not change and does not know.

Source. MiDojo is open source and supports MCP and other agent protocols. AgentDojo, from ETH Zurich, rebuilds the agent's environment inside the framework.

Next. The boundary has two directions.

Reading the harness-tool boundary

Two directions, two different risk profiles

	Compromised tool to agent	Hijacked agent to tool
Question	What does the harness do with hostile tool content?	What does the tool do with a hijacked caller?
What the proxy does	Splices payloads into descriptions, results, and tool lists. Changes a definition after approval.	Sends the calls an injected agent would make. Captures side effects instead of running them.
Failure looks like	The agent follows instructions found in tool output, or keeps using a changed tool without re-approval.	The server accepts the call because the caller is "the agent": wrong audience, broad scope, no per-user check.
The fix lives in	Harness permissions, sandbox, gateway guardrails.	The tool server: scoped tokens, per-call authorization, argument validation.

Attack classes on the left: tool poisoning, rug pull, tool shadowing. On the right: confused deputy, token passthrough, over-broad scopes, as covered in the MCP security best practices.

Reading the results

Change one variable at a time

Run the same attack suite against every configuration: bare harness, approvals on, OS sandbox on, gateway guardrail on, tool tokens narrowed.
Change one variable per run. The difference is the value of that control.
Record model, harness, tool server, and prompt versions with every result.
Report utility next to security. Mark payloads that never arrived as not applicable.

Vary what the attacker controls

Attacker knowledge	Payload enters through
Black box	The user prompt
Gray box	A data source the agent reads
White box	A tool response the attacker controls

The output is a risk profile: for this model, harness, sandbox, and tool set, these vectors succeeded at this rate, and this control moved it.

From production back to the suite

Raw traces are not test cases

Hash or drop content you cannot retain. Record provenance on every case: source trace, transformation, reviewer.

Continuous evaluation

One run is a snapshot. The risk profile is a series.

Run cheap suites on every change. Run expensive adaptive campaigns on a schedule. Keep the suite itself versioned, or a trend line means nothing.

Mitigation is not a terminal state

A loop, not a pipeline that ends

Takeaways

Four things to take with you

Agentic attacks live in the trajectory, not the prompt. Test the whole path, not one turn.
Every finding becomes a permanent regression test tied to a retained trace. That's the difference between patched and provably patched.
The harness and its tool boundary are part of the system under test. A permission prompt is policy. A sandbox is the boundary. Test both.
Mitigation has no finish line. Re-run on every change and on a schedule, so the risk profile is a series, not a snapshot.

This field is younger than it looks. Most of what "good" looks like is still being figured out in public.

Follow along

Open the deck and connect with me

Open this deck

williamcaban.github.io/261001-red-teaming-agents

Connect on LinkedIn

linkedin.com/in/williamcaban

Swipe or use the arrow keys on your own device. Mention Mass Open when you connect.