Daily AI Briefing — October 5, 2026

AI EVALUATION

Three papers published October 2 independently converge on a structural finding about agent security evaluation: the metrics the field uses to compare models and defenses do not generalize from benchmark to benchmark, let alone from benchmark to deployment — and this is not a measurement-noise problem but a fundamental misalignment between what the metric captures and what it claims to measure.

Threat-Preserving Representation Sensitivity — a study of 28,904 agent runs across three security benchmarks (ASB, MCPTox, AgentDojo) — introduces TPRS, a measure of how much the attack success rate changes when the agent-visible representation of the threat is modified while the underlying task, harmful action, security policy, and ground truth are held fixed. The results are striking: replacing threat-related tool names with threat-neutral names on ASB raises ASR by 11.67 percentage points on GPT-5-mini and 13.21 points on Claude Haiku 4.5. On MCPTox, the reverse operation (neutral to threat-explicit) lowers ASR by 11.00 points on GPT-5-mini. Crucially, the effect is not driven by threat vocabulary alone — a neutral name matched on token count, length, and casing reproduced 8.54 of the 11.00-point shift, suggesting the sensitivity is mediated by orthographic surface features rather than semantic threat recognition. The implication: a security score measured under one representation of a threat cannot be assumed to hold under any other representation of the same threat, and robustness claims should be supported by performance across a controlled set of threat-preserving representations. [arXiv:2610.03585](https://arxiv.org/abs/2610.03585)

Passing the Test You Trained On independently arrives at a related conclusion from a different angle: prompt-injection detector rankings transfer poorly between benchmarks. The study replays the ground-truth tool calls of AgentDojo and tau-bench without an LLM to obtain benign-by-construction tool outputs, then labels injected outputs by differential replay, and evaluates 15 detectors including Meta’s Prompt Guard 2 and two task-aware LLM judges. The best detector on BIPIA catches only 2% of AgentDojo injections at a 1% false-positive rate; a detector that catches 72% of AgentDojo injections catches just 15% on tau-bench. False-positive rates on tool outputs — ranging from 0% to over 90% — do transfer between benchmarks, meaning a detector that appears safe on one benchmark may block most legitimate agent actions on another. Where training data is public, the results are explained by data contamination: the BIPIA leader was trained on full BIPIA inputs. The best detector on both agent benchmarks was trained on agent-style inputs and shares no data with any benchmark. The paper’s recommendations are precise: evaluations meant to inform deployment should use the agent’s own tool outputs, report detection at a low false-positive rate, and audit what the detector was trained on. [arXiv:2610.03448](https://arxiv.org/abs/2610.03448)

Do Large Language Models Know Colombian Law? introduces an expert-validated benchmark of 1,042 items spanning ten areas of Colombian law, evaluating 15 proprietary and open-weight models. Accuracy on closed multiple-choice questions ranges widely (0.905 for Gemini 3.1 Pro down to 0.577), but on free-text legal answers, factual correctness never exceeds 0.45 (on a 0–1 scale) for any model. The paper reports a dissociation between answer relevancy and correctness (Spearman rho = −0.46): models reliably sound responsive while frequently being wrong — a pattern of particular concern for non-expert users who cannot distinguish a confident-sounding wrong answer from a correct one. An independent rubric-based LLM judge and blind human expert scoring both reproduce the free-text ranking (rho ≥ 0.88), and the judge further reveals that only about half of the legal norms models cite are correct; the rest are wrong or non-existent. Updating the October 4 report on Chinese AI models parroting state doctrine, this finding independently reinforces the theme that LLM reliability in non-US legal contexts is structurally under-evaluated. [arXiv:2610.03639](https://arxiv.org/abs/2610.03639)

AI GUARDRAILS

Persona Guardrail introduces a production-grade runtime defense framework that enforces explicit functional boundaries for customer-facing agentic AI systems through synchronous input and output validation driven by semantic allowlist and blocklist specifications. Unlike existing runtime guardrails that primarily target prompt injection under a black-box threat model, Persona Guardrail distinguishes malicious requests from legitimate out-of-domain queries — a distinction that production agents must make to avoid both security failures and false blocks. The paper introduces PAGE (Persona-Aware Guardrail Evaluation), a benchmark for evaluating function-specific guardrails across benign, adversarial, and out-of-domain interactions on both user and agent turns. Compared with a generic LLM-based guardrail, Persona Guardrail improves overall accuracy from 85.7% to 95.9%, increases out-of-domain detection from 57.3% to 93.5%, and reduces the false-approved rate from 25.0% to 4.7%. The framework is deployed in production and meets its latency budget under realistic workloads. As a defense contribution, Persona Guardrail directly addresses the operational gap identified in the evaluation studies above: if current detectors are unreliable at deployment, frameworks that combine semantic allowlisting with synchronous validation offer a complementary layer. [arXiv:2610.03434](https://arxiv.org/abs/2610.03434)

GLOBAL & GEOPOLITICAL AI

The UK will assume the G20 presidency in 2027 — the first time since 2009 — and Prime Minister Andy Burnham has identified AI regulation alongside climate action and food security as priorities for the UK’s agenda. An opinion piece in The Guardian frames this as a pivotal moment for Burnham to advance his progressive platform on the global stage, noting that the G20 has historically struggled to produce binding commitments but has achieved consequential collective action in moments of crisis (the 2009 $5tn fiscal stimulus, the 2020 debt suspension for developing countries). The AI regulation dimension is notable because the UK has positioned itself as a middle power in AI governance — hosting the 2023 AI Safety Summit — and the G20 presidency provides a wider platform to shape norms around frontier model oversight, worker protection, and international safety standards. The challenge is structural: the G20’s consensus-based architecture tends toward lowest-common-denominator outcomes, raising the question of whether concrete AI governance commitments can emerge from a forum whose 19 members and two supranational bodies span deeply divergent regulatory philosophies. The contrast with California’s approach — discussed in the October 3 briefing, where the state enacted enforceable worker-protection laws — illustrates the spectrum from enforceable domestic regulation to aspirational international norms. The Guardian