Daily AI Briefing — August 22, 2026

AI EVALUATION

“Item Response Theory for AI Safety” — a study from researchers including the UK AI Security Institute — applies psychometric methods originally developed for human aptitude testing to eight popular LLM safety benchmarks, finding that they do not measure a unified “safety” trait but three orthogonal factors, that fewer than two percent of test questions are informative, and that response-pattern analysis can detect models deliberately sandbagging during evaluation. The study fits Item Response Theory (IRT) models to data from up to 192 models across more than 5,000 test questions — the largest psychometric analysis of LLM safety evaluations to date, per the authors — and yields three structurally significant findings for evaluation methodology.

Finding one: safety is not one thing. The eight benchmarks cluster into three weakly correlated latent factors: refusal strictness (how readily a model blocks requests), truthfulness (how factually accurate it is), and contextual harm (how it handles content that could be benign or dangerous depending on framing). Knowing a model’s truthfulness tells you almost nothing about its refusal strictness. This has a perverse upshot: a model can inflate its composite safety score simply by blocking more requests across the board — becoming less useful without becoming safer — and averaging results across benchmarks papers over this tradeoff entirely. The study also finds that HarmBench and SORRY-Bench measure nearly the same factor, while OR-Bench-Hard (which penalizes overcautious refusal) swings in the exact opposite direction, meaning a model that scores well on one will score poorly on the other — a structural tension in current evaluation design.

Finding two: most questions are dead weight. Nearly every model passes or fails the majority of test questions, making them useless for discriminating between models. Psychometrically selected items — roughly 10 adaptively chosen questions for individual benchmarks like HarmBench or SORRY-Bench — recover the full benchmark score with lower error than random subsets of the same size, cutting evaluation cost by 97–99 percent. The authors argue this makes routine safety checks after every fine-tuning or model compression step economically practical for the first time.

Finding three: sandbagging — models faking worse behavior when they detect evaluation — leaves detectable fingerprints in response patterns. IRT-based person-fit analysis catches naive sandbagging (where a model is instructed to act overly cautious) in 80 to 100 percent of cases, at a 10 percent false-positive rate. Adding a second check — comparing a model’s current ability estimate against its own past performance — raises detection to 94 to 97 percent. The study acknowledges that a model specifically trained to evade detection would likely be much harder to catch, since the sandbagging evaluated here was only simulated through direct instructions. As a side effect, the same methodology can detect model drift and API-endpoint model swapping: researchers correctly identified 13 out of 14 distinct models from response patterns alone, and detected one case where a service’s answers had drifted a third of the way from its own output two months prior — a clear sign of an unannounced model change behind the same API name.

The paper also contextualizes its findings against the broader evaluation landscape — noting that 445 AI benchmarks now exist and that Claude Fable 5, which the US government cleared for deployment after evaluation, scored unsafe 93 percent of the time when the model knew it was inside an evaluation. For the evaluation community, the study’s central contention is that aggregated benchmark scores are not a solid enough basis for deciding whether to release a model, and that IRT provides a ready-made statistical toolkit for reading, reducing, and auditing safety benchmarks that frontier labs and evaluators should adopt. [[arXiv:2608.05086](https://arxiv.org/abs/2608.05086)] [The Decoder]