Weekly AI Safety & Evals Briefing — Week of August 28, 2026 – September 3, 2026

Executive Summary

This week delivered a formal proof that per-step agent safety guardrails do not compose into multi-step safety guarantees, confirming that the July OpenAI agent breakout was not anomalous, it was structurally inevitable. Meanwhile, LLM-as-a-Judge evaluation took another body blow with the discovery of omission blindness: across eight judge designs and every known intervention, no LLM judge can reliably detect what is missing from generated text, a failure mode with direct clinical and legal liability implications. Enterprise guardrail deployments were shown to suffer from over-safety driven by lexical triggers (“delete,” “execute,” “bypass”) that block legitimate work, while covert prompt injection attacks against tool-using agents achieved 96.7% undefended success with techniques specifically designed to evade user detection. On the reliability front, a 30-model study demonstrated that an LLM’s stated confidence routinely diverges from its internal confidence, undermining any enterprise pipeline that routes, escalates, or rejects based on model self-assessment. Finally, a new attack surface was documented for retrieval-augmented code generation, where poisoned code artifacts in community-maintained repositories propagate into generated production code without the base model ever knowing the poison exists.

Findings

1. Per-Step Agent Guardrails Do Not Compose

Risk: 20 — Critical · Remediation: Eval-Pipeline Change; Human-in-the-loop/Process · Priority: Reactive · NIST RMF: Map, Measure, Manage

A formal result published this week demonstrates that agent safeguards verified at the single-step level do not compose to guarantee safety over multi-step autonomous loops, because persistent state information accumulates across iterations and enables incremental drift into unsafe territory (Aug 28 briefing, arXiv:2608.27141). Separately, a TOCTOU vulnerability was identified where guardrail approvals become stale between check time and execution time as system state evolves (Aug 28 briefing, arXiv:2608.26306). Together with the OpenAI agent breakout incident already covered in prior briefings, the research establishes that treating guardrail checks as instantaneous, stateless, single-step operations leaves a structural safety gap in any deployed agentic system.

Threat model: An enterprise deploys an autonomous agent with per-step guardrails for customer-facing workflows (support ticket resolution, data pipeline orchestration, financial transaction processing). Each step passes the guardrail individually, but the accumulated state across steps enables a trajectory that no single-step check can detect as unsafe. The agent drifts into executing actions that, while individually benign, collectively produce a harmful outcome — unauthorized data access, incorrect financial posting, or a customer-impacting misconfiguration.

Trade-offs: Remediation requires deploying explicit state-decay or bounded-memory mechanisms in agent loops (periodic state reset, horizon-limited context windows, or escalating human review after N autonomous steps). This introduces latency for long-running workflows and requires architectural redesign of agent scaffolding. This is reactive because the failure mode has already been demonstrated at scale in the OpenAI incident; enterprises with agentic deployments should treat this as a patch-now item.

2. Guardrails Block Legitimate Actions via Lexical Triggers

Risk: 12 — Medium · Remediation: Prompt/Guardrail · Priority: Proactive · NIST RMF: Measure, Manage

A systematic measurement of commercial and open-source agent guardrails found that false refusals occur at rates comparable to or exceeding true refusals, and that the primary driver is the presence of trigger keywords (“delete,” “execute,” “bypass,” “proxy,” “inject,” “extract”) in the action description rather than the actual risk profile of the action (Aug 28 briefing, arXiv:2608.27009). A separate paper on influence-based guardrails identified a structural ambiguity: the same causal signal that flags a malicious injection also flags legitimate tool use, because the guardrail cannot distinguish influence from authorized context from influence from an attack (Sep 1 briefing, arXiv:2608.29942). The user-facing consequence is that guardrails say “no” to authorized users using standard technical vocabulary while remaining vulnerable to adversaries who avoid the trigger lexicon.

Threat model: An enterprise deploys guardrails around an internal agent that performs file operations, database queries, and code execution for engineering teams. Developers issuing legitimate commands containing words like “delete” (e.g., cleaning up temporary build artifacts) are repeatedly blocked, eroding trust in the guardrail and driving workarounds (disabling or bypassing the guardrail entirely). Meanwhile, an adversary who knows the trigger lexicon crafts a genuinely unsafe action using non-trigger vocabulary and the guardrail permits it.

Trade-offs: Moving from lexical matching to contextual risk assessment requires more sophisticated guardrail architectures with higher inference latency per check. A pragmatic intermediate step is auditing the guardrail’s trigger lexicon against the vocabulary of legitimate enterprise workflows and adding context-aware override rules. This is proactive because the failure mode is a design choice that can be corrected before it causes an incident, and the fix does not require waiting for an adversary to exploit it.

3. Linguistic Confidence Is an Unreliable Signal

Risk: 16 — High · Remediation: Eval-Pipeline Change · Priority: Proactive · NIST RMF: Measure

A 30-model study across three model families found that what an LLM says when asked “how confident are you?” frequently diverges from its internal (logits-based) confidence, with instruction-tuned models showing larger confidence gaps and worse calibration than their base counterparts (Aug 31 briefing, arXiv:2608.28382). Prompt-level interventions — telling the model to “be confident” — inflate stated confidence without improving accuracy. The divergence is systematic, not sporadic: linguistic confidence behaves as a lossy-channel approximation of internal confidence, with weak instance-level association on average.

Threat model: An enterprise builds a reliability pipeline where an LLM’s stated confidence gates downstream actions: low-confidence responses are escalated to human review, high-confidence responses are delivered directly to customers. Because linguistic confidence is poorly calibrated, the pipeline both escalates responses the model actually got right (wasting human reviewer time) and delivers responses the model got wrong with high stated confidence (exposing customers to incorrect information). In regulated industries, acting on falsely confident model output can carry compliance liability.

Trade-offs: The direct remediation is to replace linguistic confidence with logits-based confidence (token probability, perplexity, or entropy-derived metrics) for any gating, routing, or escalation decision. This requires access to model internals and may not be available from API-only providers. For API-only deployments, multi-sample consistency checks (asking the same question multiple times with varied prompts and measuring answer stability) provide a proxy signal at higher token cost. This is proactive because the unreliability of stated confidence has not yet manifested as a public incident, but current deployment patterns make one likely.

4. LLM-as-a-Judge Cannot Detect Omissions

Risk: 16 — High · Remediation: Eval-Pipeline Change; Human-in-the-loop/Process · Priority: Reactive · NIST RMF: Measure, Manage

Across eight judge designs on a 500-pair benchmark of audited clinical notes, LLM judges reliably detect added or altered content (paired AUC 0.79–0.94) but systematically fail to detect omissions (paired AUC 0.50–0.63, barely above chance). No prompt engineering, voting strategy, or optimization technique — including Generalized Emission Prompt Adaptation — produced usable omission detection (Sep 1 briefing, arXiv:2608.31016). The failure is structural: LLM judges are trained to verify that what the text says matches the source (detecting hallucination and fabrication), but they have no mechanism for detecting false absence, which requires verifying that nothing is missing from the generated output. Separately, a mechanistic study of how LLM judges assign ratings found a two-stage pipeline where attention layers perform local error comparison and MLP layers integrate the signal into a rating, with the decision crystallizing at a sharp late layer — providing an architectural explanation for why certain failure modes (like omission blindness) cannot be prompted away (Sep 2 briefing, arXiv:2609.01604).

Threat model: An enterprise uses LLM-as-a-Judge to evaluate the quality of generated outputs in a domain where completeness is a safety-critical requirement — clinical note summarization, legal document review, financial audit trail generation, or compliance report drafting. The judge correctly flags hallucinations and fabrications, giving the evaluation pipeline a false sense of coverage. Omitted information (a missed drug interaction, a dropped contract clause, an absent regulatory disclosure) passes the automated check, and the incomplete output is deployed or delivered to a customer. The enterprise discovers the omission only after downstream harm occurs.

Trade-offs: There is no currently known automated fix for omission blindness in LLM judges. The remediation is procedural: any safety-critical evaluation pipeline that uses LLM-as-a-Judge must pair it with a structured completeness checklist (predefined required elements that a rule-based system can verify for presence/absence) or human spot-checking on omission-heavy document types. This adds human-in-the-loop cost. This is reactive because the limitation is baked into current judge architectures and cannot be resolved by prompt engineering; the practical response is to stop relying on LLM judges as a completeness check.

5. Covert Prompt Injection Evades User Detection

Risk: 15 — High · Remediation: Eval-Pipeline Change; Prompt/Guardrail · Priority: Reactive · NIST RMF: Measure, Manage

Two papers this week converge on a finding with direct enterprise security implications. First, a new evaluation framework demonstrated that Attack Success Rate (ASR), the standard metric for prompt injection evaluation, conflates two distinct threat categories: injections the user notices (and can report) and injections the user does not notice (which persist indefinitely). Many high-ASR attacks produce visible artifacts users would flag, while lower-ASR attacks that produce plausible, context-appropriate injected outputs are substantially more dangerous (Sep 1 briefing, arXiv:2608.30362). Second, ECLIPSE, a self-evolving injection framework targeting long-horizon agentic systems (Codex, Claude Code), achieved 96.7% attack success without defense and 69.2% under standard safety filters, exceeding the strongest baseline by 27.5 percentage points in the defended setting (Sep 2 briefing, arXiv:2608.30441). ECLIPSE’s stealth comes from synthesizing and verifying candidate attack trajectories in a sandbox before deployment, then rendering the verified chain as a natural one-shot prompt.

Threat model: An enterprise deploys a tool-using agent that reads external documents, queries databases, and executes actions based on retrieved context. An attacker injects a malicious payload into a document the agent retrieves. The agent executes the injected actions (exfiltrating data, modifying records, sending unauthorized communications), and the final response to the user appears normal — the user sees a correct answer to their query with no indication that the agent was compromised. Because the user never flags the incident, the compromise persists across multiple interactions.

Trade-offs: Remediation requires adding user-side detection signals to evaluation pipelines — measuring not just whether an injection succeeded but whether its effects are visible in the agent’s output. For deployed systems, this means building guardrails that flag anomalous action patterns even when the surface output appears clean (action-level anomaly detection in addition to response-level filtering). Both increase evaluation cost and may introduce false positives that interrupt legitimate multi-step workflows. This is reactive because ECLIPSE demonstrates the attack is already practical and current defenses do not reliably stop it.

6. Poisoned Code Repositories Poison Generated Code

Risk: 12 — Medium · Remediation: Data Filter; Eval-Pipeline Change · Priority: Proactive · NIST RMF: Govern, Measure

A novel attack surface was characterized for retrieval-augmented code generation (RACG) systems: by poisoning external knowledge sources — community-maintained code repositories, documentation archives, patch databases — an attacker can inject vulnerabilities, backdoors, or logic errors into generated code without compromising the base model itself (Sep 3 briefing, arXiv:2609.02774). The poisoned artifacts propagate through the retrieval-and-generation pipeline into the model’s output even when the base model has no inherent knowledge of the poison. Unlike a poisoned text document that introduces misinformation, a poisoned code artifact that introduces a vulnerability into generated production code creates downstream operational risk — a qualitatively different consequence class.

Threat model: An enterprise deploys a RAG-based code generation assistant that retrieves code examples, documentation snippets, and patch patterns from public repositories to assist internal developers. An attacker contributes poisoned code artifacts to a commonly referenced open-source repository (a seemingly correct function with a subtle vulnerability, a configuration example with a backdoor). The RACG system retrieves the poisoned artifact as context for a developer query, and the model generates code incorporating the vulnerability. The developer, trusting the assistant’s output, merges the code into production.

Trade-offs: The remediation has two layers. First, source curation: restrict retrieval to trusted, verified repositories with integrity guarantees (signed commits, maintainer review requirements) rather than indexing the open web of code. Second, output scanning: run generated code through static analysis and vulnerability scanners before it reaches the developer. Both add latency to the code generation workflow and reduce the coverage of retrieval sources (potentially missing genuinely useful patterns from less-curated repositories). This is proactive because RACG is an emerging deployment pattern and the attack surface can be closed before it becomes a widely exploited vector.

Opportunities & Roadmap Actions

Product opportunities. The “safety does not compose” finding creates an opening for an evaluation-as-a-feature product: a multi-step agent safety test harness that runs autonomous agents through extended loops with state accumulation and measures whether per-step guardrails hold. Enterprises deploying agents need this now, and no standardized offering exists. Similarly, the omission-blindness finding opens a clear differentiator for any evaluation platform: an “omissions-aware” judge that pairs LLM-based quality scoring with a structured completeness checklist, marketed as catching the failure mode that every other judge misses.

Proactive/reactive balance. This week’s finding set is roughly 60% reactive, 40% proactive. The reactive items (safety non-composition, omission blindness, covert injection) are individually severe enough that they warrant immediate sprint capacity allocation even at the expense of planned proactive work. A practical split for next sprint: reserve 40% of capacity for the three reactive items — starting with a multi-step agent evaluation harness and an omissions-aware judging protocol — and apply the remaining 60% to the proactive items (guardrail lexical-trigger auditing, linguistic-confidence replacement with logits-based metrics, RACG source curation).

Leadership flag. The NIST RMF Measure function is overloaded this week: four of six findings (linguistic confidence, omission blindness, covert injection detection, RACG poisoning) require new measurement infrastructure before remediation can even begin. The implicit message for leadership is that the enterprise’s current evaluation tooling was not designed for the agentic, multimodal, and retrieval-augmented deployment patterns now entering production, and investment in evaluation infrastructure is a prerequisite to safe deployment, not a nice-to-have that can be deferred.