Daily AI Briefing — November 8, 2026

AI EVALUATION

A new coding-agent benchmark built from requests that real users already struggled to get agents to handle finds that even the best model fully completes about one task in four. Co-Design Bench (paper at arXiv:2610.35138, benchmark site at codesignbench.github.io, per The Decoder’s coverage) comes from Alibaba researchers and collaborators, who mined actual coding sessions, deliberately favoring cases where users had to rephrase requests multiple times or roll back changes. Ten models were tested, including multimodal tracks where agents must extract values from receipts, tables or videos and update every dependent view in a web app. GPT-6 Astra leads both tracks but meets only about 70 percent of a task’s requirements on average and is fully correct on 26.4 percent of tasks, with Claude Opus 5 and Qwen3.8-Max behind it and Grok 4.6 last. The authors’ motivation is saturation: on older tests like WebCompass-edit, agentic coding models exceed 99 percent, a gap of more than 35 percentage points versus Co-Design Bench. Two methodological points travel beyond coding. Grading is adversarial, one AI judge tests the finished app in a browser and a second re-checks every positive verdict while hunting for mock-ups and hardcoded results, and a controlled harness comparison found the same model (Qwen3.8-27B) scoring 75 or 56 percent on large overhauls depending on the working environment, with model rankings reversing between harnesses, so leaderboards that report the model alone tell part of the story. Stated limitations: one fixed request per task, no mid-task user corrections, and AI judges that can themselves err. Extra compute did not close the gap, GLM-5.2 burned nearly six times as many tokens as GPT-6 Astra on smaller changes and still scored significantly worse, and all ten models degrade as tasks combine three or four types of change, which is the norm in real projects.

A training-free self-evaluation method reads correctness from how attention and feed-forward components interact across layers, rather than from output signals alone. Circe (arXiv:2611.07948, Miao, Li, Xie, Xu, Shen, Du and Chua, submitted November 5) targets reference-free evaluation, where existing methods rely mainly on output signals that can assign high scores to wrong answers. The authors argue that prior internal-state work analyzes hidden-state trajectories across layers, which record only the combined updates of attention and feed-forward networks and miss how the components interact. Circe measures component interaction consistency, the directional alignment between attention updates in earlier layers and attention-induced FFN output changes in later layers, adds representation-geometry features along the layerwise trajectory and a bounded likelihood-confidence score, then fuses the three into one correctness estimate, with no reference answers and no repeated sampling. Across five LLMs and six benchmarks in four domains the paper reports better discrimination of correct from incorrect responses than existing baselines, with generalization to multilingual, multimodal and open-world settings. The usual caveats for this genre apply: the evaluation covers the authors’ own selection of benchmarks, the method was validated on five models, and no adversarial responses were studied, so the claim is relative performance on standard distributions rather than robustness.

A JMLR paper offers a rigorous answer to a question that keeps recurring in AI-system auditing: what can observational data certify about a simulator? “Causal Falsification of Digital Twins” (Cornish, Taufiq, Doucet and Holmes, JMLR 27(152):1-52, 2026) formulates digital-twin accuracy as a causal inference problem and, drawing on fundamental results in that field, concludes that observational data alone cannot certify a twin as correct unless tenuous assumptions such as unconfoundedness hold. The constructive move is to search instead for situations where the twin is demonstrably wrong, under only the assumption of an i.i.d. dataset of trajectories, so the procedure stays sound even with confounded data. The case study applies the method to sepsis modeling in the Pulse Physiology Engine using the MIMIC-III ICU dataset, and the code is public. The relevance to trustworthy AI is direct: the same logic bounds what any evaluation built on logged deployment data can prove about a generative model used to simulate interventions, certification claims require experimental or identification assumptions, and falsification-oriented testing is the fallback that remains valid.

AI SAFETY & ALIGNMENT

A Vals AI benchmark of autonomous network intrusion widens the gap between vendor-controlled and freely available models into a governance problem, and its runtime finding undercuts how the capability is usually measured. Per The Decoder’s coverage of the CyberOffense benchmark, each agent operates alone on an isolated attack machine with nothing but a public web address and zero human help. GPT-6 Sol, from OpenAI’s restricted Daybreak cyber program, achieved 80.8 percent of objectives on the main corporate-network simulation; DeepSeek V4.1 Flash 1017, a freely downloadable open-weight model, reached 72.8 percent at roughly $3 per run against about $22. On the harder Lighthouse scenario, Daybreak took admin control of a first company, stole passwords and reused them to break into a second, completing 27 of 31 objectives in under three hours, while DeepSeek managed 21 at a tenth of the cost and was, per the report, not limited solely by safety guardrails. The methodological finding matters for any cyber eval: DeepSeek scored zero in its first 25 million tokens and made most of its progress only after 100 million, so short test runs systematically underestimate open models, a variant of the harness-dependence pattern seen elsewhere this week. The results align with the UK AI Security Institute’s July analysis that the open-versus-closed frontier gap in cyber capability had shrunk to four to seven months, down from six to ten for most of 2025, with AISI also reporting that safety measures on the open models proved largely ineffective and citing a $1.19-per-run cost for DeepSeek V4-Pro against $85 for Anthropic’s Opus 4.5. Anthropic’s own study found the open GLM-5-3 building exploits at nearly the level of its internal top model. Vals’ framing cuts both ways: once weights are public no provider can restrict use, and the same models can help defenders find and fix vulnerabilities first.

AI GUARDRAILS

An analysis of Thinking Machines’ Inkling 2 Small found profanity throughout its internal reasoning, and the more substantive result is that the visible signs of a failing run carry almost no signal. Per The Decoder’s report on Vals AI’s analysis, reading chain-of-thought logs from 2,655 runs across 13 benchmarks found expletives in 50 runs, 316 instances in total, with 241 coming from just two runs that spiraled: in one, a task to calculate an asteroid’s trajectory produced a single 500,000-character response, roughly novel-length, repeating the same sentences with 158 expletives. The model typically swears at its own mistakes, apparent breakthroughs or when giving up, and one run read “FUCK. SUBMIT. DONE. IT’S NOT WORKING. NOTHING WORKS.” The null result is the point: 46 of 48 scored runs containing profanity failed, but about 75 percent of all scored runs fail anyway, and 96 percent of failed runs contained no expletives, so profanity is not a usable warning signal. Users typically never see these thinking steps, or see them only through a filter, which matters for anyone building monitoring or anomaly detection on top of CoT streams: surface features of the reasoning channel can be uninformative precisely where they look most alarming.

GLOBAL & GEOPOLITICAL AI

Tencent’s chief AI scientist publicly warned that frontier AI will soon be powerful enough to master biology, adding a major Chinese lab’s voice to the biosecurity-risk conversation. Per SCMP’s report from November 8, Vince Yao Shunyu, Tencent Holdings’ chief AI scientist and a former OpenAI researcher, said during a rare public appearance on Saturday that AI models will soon be capable of mastering biology, presenting both benefits and serious risks, as frontier systems continue to scale. The remark is notable as positioning rather than as a technical disclosure: no supporting evidence or capability data was offered in the coverage, and comparable warnings from US frontier labs earlier this year were likewise framed as anticipatory risk assessment. It does, however, align with the direction of published cyber-capability trajectories covered above, where the interval between restricted-program results and freely reproducible open-weight results has compressed to months, and the same publication pattern would plausibly apply to wet-lab-adjacent capability claims.