Voxtral 3

Mistral released new speech-to-text and text-to-speech models plus a voice agent SDK, built on LiveKit, that streams transcription, an LLM with tool calls, and speech output as one pipeline. Mistral reports turns under one second, transcription in 50+ languages at $0.0035/min for the Mini model, and speech at $13 per 1M characters, on its EU-hosted API or self-deployed. The benchmark charts are Mistral’s own.

LLMs are making formal software verification feasible at a much larger scale

Anthropic and Theorem had Claude prove that 239 fuzz entry points in 123 OSS-Fuzz libraries never crash for any input under 64 MB, producing about 180 million lines of Rocq over compiled RISC-V binaries. The work surfaced bugs that fuzzing had missed. The proofs assume single-threaded static RISC-V binaries on Linux with musl, while most deployments run x86 or ARM with glibc.

OpenAI models are coming to Google Cloud

OpenAI’s announcement says its frontier models and Codex workflows are coming to Model Garden, sold through Google Cloud Marketplace. Enterprise clients that buy through Google Cloud contracts can add OpenAI models without a separate vendor agreement. The announcement summary gives no model list, pricing, regions or data-residency terms, so those questions stay open for procurement teams.

Patch the Planet: global response coordination as agents uncover critical vulnerabilities

OpenAI launched Patch the Planet, a program connecting security researchers, maintainers and incident responders to fix critical open-source vulnerabilities as AI agents find them. The announcement describes a coordination program and reports no counts of vulnerabilities handled. Read next to Anthropic’s verification work, it suggests labs expect AI-found bugs to arrive faster than maintainers can triage them.

LEV: Thinking Fast and Deep with Looped Decision Models

What problem does it solve?

Decision models such as Jev and Kev answer questions with a fixed set of options by returning one probability per option from a single forward pass, in tens of milliseconds. Their depth is fixed at training time, so an easy question costs as much as a hard one. On the 231 public JevBench items, Kev-4B answers 28 more items correctly than Kev-0.8B, all from the standard and hard tiers. Generated reasoning on a DiffusionGemma 26B base raises hard-tier accuracy by 9 to 13 points but makes the median hard item three to four times slower, and easy questions pay for reasoning text too. Cascades reread uncertain questions and need a second hosted model. A loop retrofitted onto Qwen3.5-0.8B moved accuracy by less than half a point.

How does it solve the problem?

LEV, from Lehigh University with the University of British Columbia and the Vector Institute, starts from language models pretrained as loops: Ouro-1.4B and Ouro-2.6B, which rerun their layer stack four times, and Nanbeige4.2-3B, which runs it twice. A shared pointer head scores the options after every loop step, and training averages the loss over all steps with a rank-16 LoRA adapter. One checkpoint answers at any depth, depth k costs about k/R of the full pass, and an extra step resumes from the previous hidden states without rereading the input.

The usual early-exit rule is a confidence threshold tuned on labelled data. It fails here because shallow readouts are overconfident on new data: questions LEV-2.6B stops at depth one are 94.8% correct on the calibration partition but 78.9% correct on Kev-Transfer at almost the same confidence. LEV’s stopper, a small per-depth logistic regression, instead predicts whether the current answer agrees with the model’s own deepest answer. Its threshold is the cheapest one that keeps at least 98% agreement on unlabelled requests. About 500 unlabelled questions per fold bring LEV-2.6B within 0.2 points of the full fit.

What are the key findings?

• LEV-2.6B rises from 63.2% at depth one to 76.2% at depth four on JevBench with the same weights, level with the released Kev-4B at 75.8%.

• The stopper spends 1.2, 2.4 and 3.0 steps on easy, standard and hard JevBench items without seeing the tier.

• On-demand depth saves 15% to 24% of computation for under one point against the deepest readout. Against a fixed mix of depths at equal cost, the locked-test gain is +0.1 points for LEV-2.6B and +0.9 for LEV-3B.

• Below 50 ms, LEV-1.4B and LEV-3B are 5.4 to 8.8 points more accurate than dense models and cascades of the same latency. Part of this comes from no dense option answering between 22 and 79 ms, and in FLOPs LEV is at or below the dense options, not ahead.

Cost is counted in loop steps and latency was measured one request at a time, so batching mixed depths may shrink the wall-clock saving. On LevBench, 12.6% of questions are right at a shallower step and wrong at the deepest, which an agreement-based stopper cannot recover.

Why does it matter?

For fixed-option work such as routing, policy checks and classification into enums, LEV turns depth into a per-request setting on one checkpoint instead of a choice between two hosted model sizes. The stopper recipe also applies elsewhere: fit early exit to agreement with your deepest model on unlabelled production traffic, since labelled calibration sets drift from live data. Count the gain as latency, not FLOPs. The code is public on GitHub under Apache-2.0. LEV is the subject of this week’s video and deep dive, both published on Wednesday.

Has GPT-6-Astra Achieved Embodied Intelligence?

What problem does it solve?

Robotic manipulation and navigation have been the domain of specialized models trained per task and embodiment, such as vision-language-action models (VLAs). A joint research team led by SenseNova tests how far a general frontier model, GPT-6-Astra, goes as a robot controller, and measures what separates its benchmark scores from use on physical robots.

How does it solve the problem?

The study has four parts. In simulation, across 12 benchmarks covering tabletop arms, dexterous hands, a humanoid, household tasks and navigation, the weights stay frozen. Before testing, the model studies each task’s training demonstrations and writes a frozen knowledge package: task guidance, motion and visual references, Python helpers for kinematics, completion checks and recovery procedures. Episodes run at the official step budget and at up to 10x. Second, harness ablations on RoboDojo pair GPT-6-Astra as a slow ‘System 2’ with a fine-tuned π0.5 VLA as ‘System 1’, either issuing subgoals open-loop or reviewing the VLA’s proposed end-effector actions before execution. Third, zero-shot runs on 8 real robot platforms use human-written context and a 60-minute wall-clock budget. Fourth, a household prototype in NVIDIA Isaac Sim keeps the world clock running while the model deliberates.

What are the key findings?

• In simulation, it reaches 81.9% on RoboDojo and 87.3% on DexJoCo at 10x budget, with LIBERO and SIMPLE above 95%. RoboDojo success roughly triples with the extended budget, and in-context learning lifts SIMPLE from 70.8% to 90.8% at standard budget.

• On LIBERO-Long++, where standard-budget success already exceeds 97%, each episode still takes about a million tokens and nearly an hour. The authors estimate 1M tokens at about 1 USD.

• Closed-loop review reaches 23.61% against 5.83% open-loop, at 7.664M tokens against 0.387M. Reviewing four actions per call cuts tokens by 63.5% and lifts success to 25.83%. Adding in-context learning reaches 54.17% at 19.335M tokens. Joint-angle control loses 5.84 points against end-effector targets.

• On real robots, rigid pick-and-place succeeds 17/18 times, cloth folding 2/4, navigation and search 9/23, and plug insertion 0/11. About 90% of each Unitree G1 humanoid run is deliberation. Dexterous grasping went from 0/3 to 3/3 only after a human wrote grasp primitives.

• In the household prototype, a decision to turn off a burner took 8 s of world time and the egg charred.

The knowledge packages transfer poorly. RoboDojo’s Open group, which has no task-specific demonstrations, gets the lowest success of all groups, and real tasks rarely come with demonstrations at all.

Why does it matter?

When a client cites robotics benchmark numbers for a general model, ask what the score assumed. Here it assumed demonstrations of the test task and a simulator that pauses while the model thinks, and both assumptions fail on hardware, where extra attempts trigger safety stops and overheating. For harness builders, the ablations give concrete defaults: expose end-effector or directional commands rather than joint angles, leave precise motion to low-level controllers, and review VLA proposals in chunks rather than per action.

Read more

• Google DeepMind launched Gemini 4 Argon at $2/$10 per million tokens, and Anthropic priced Claude Haiku 5.5 at about 75% less than Haiku 4.5 on average.

• VideoJev scores video answers instead of generating them, which makes multiple-choice video QA more than 100x faster at near-Thinking accuracy.

• OmniCompass tested 50 models: strong …

Read more

This week in LLM Watch:

• Base models with a light harness reach more agentic tasks than their post-trained versions once given enough rollouts, and an 8-rollout estimate predicts how much coverage post-training cost (Sharpening Tax).

• In ten real companion relationships, only 3.4% of sampled user messages needed anything beyond the current thread, so pooled memory benchmarks mostly measure messages that need no memory (RealCompanion).

• With the backbone fixed, switching from OpenCode to Claude Code raised the best score on unfamiliar games from 74.5 to 81.8 (Learn2Play Bench).

Let’s go!

Sharpening Tax in Post-Training

What problem does it solve?

A common hypothesis holds that RL post-training sharpens behaviors the base model already has: single-shot accuracy (pass@1) rises while the number of tasks solved across many attempts (pass@K) falls. The evidence comes mostly from math and coding. Agentic tasks could behave differently, because multi-turn tool use is widely assumed to be learned during post-training. Researchers at Meta Superintelligence Labs, the University of Wisconsin–Madison and Stanford test this on 14 open base/post-trained checkpoint pairs from Gemma-4, Ministral-3, Qwen2.5 and Qwen3.5 (3B to 35B effective parameters), across BFCL v4 multi-turn, WebShop and ACEBench, which gives 42 combinations (paper, code).

How does it solve the problem?

Base models run inside a model-agnostic harness: a plain-text prompt, a tool catalog written as Python signatures, and JSON tool-call blocks. Post-trained models use the default scaffolding with thinking mode off. Each task gets up to 128 rollouts. The Sharpening Tax sums the gap between pass@K and pass@k for every k below K, divides by the single-shot failure rate, and subtracts the post-trained score from the base score. The authors prove this area equals the expected number of failed attempts before the first success, so a positive tax means post-training made successes depend less on retries.

Raising the sampling temperature is the obvious fix for lost coverage, and the paper reports that it improves pass@K but lowers pass@1. Its alternative, posterior-tempered group sampling (PTGS), keeps discounted success and failure counts for each prompt during RL training, draws a difficulty estimate from a Beta posterior, and sets that prompt’s temperature between 1/τ and τ: hard prompts are sampled hotter and easy prompts cooler. The RL update itself is unchanged.

What are the key findings?

On WebShop, the gemma-4-31B base model passes over 85% of tasks at 128 rollouts, against 56% for gemma-4-31B-it. The rollout count at which the base model overtakes falls from above 128 for the 4B Gemma to about 3 for the 31B. Post-training cut the share of WebShop tasks solved sometimes but not always from 87.6% to 30.0% for gemma-4-31B, while never-solved tasks rose from 12.4% to 44.0%. The calibrated tax at 128 rollouts is positive in 36 of the 42 combinations, and its value from 8 rollouts on half the tasks predicts the 32-rollout tax on the other half with Spearman ρ=0.85. With PTGS, Qwen2.5-7B-Instruct trained with PPO on Sokoban goes from 46.5 to 61.1 pass@1 and from 55.0 to 69.7 pass@128. With GRPO, Sokoban pass@128 goes from 55.3 to 72.5.

Why does it matter?

If your product samples many candidates and has a verifier to pick one, the post-trained checkpoint is not the default choice at large model sizes. Run 8 rollouts of both base and post-trained versions on your own tasks and compute the tax before deciding. The paper also reports that routing tasks between base and post-trained policies by tax often beats either one alone. For clients judging fine-tuned agents, ask for pass@K and the tax next to pass@1.

RealCompanion: Benchmarking Human Understanding from Reasoning over Longitudinal Real-World Conversations

What problem does it solve?

Memory benchmarks such as LoCoMo and LongMemEval generate the conversation and write the questions, so nearly every probe needs the past and the benchmark never tests whether retrieval was warranted. Quis Lab releases ten real relationships with an AI companion: 27,218 messages over 36 to 120 days, released with consent and anonymized (paper).

How does it solve the problem?

Each participant ships with five files: the conversation and four files derived from it (profile, persona, chat ground truth, question set), with every claim citing the message identifiers it rests on. Chat probes are messages users actually sent. Labels come from a five-stage pipeline (referent, verification against the source, rule-based tier, grounded reference reply, validation) that records its reasoning at each stage. Verification can remove evidence but never add it, so labeling errors can only make memory look rarer. A random proportional sample of 1,169 scoreable probes supports rates. An enriched sweep supplies the memory-bearing cases, and 434 session openers test whether a system holds back.

What are the key findings?

On the proportional sample, 3.4% of messages need anything outside the current thread (1.3% under a strict reading). When memory is needed, the furthest required message sits a median of 2,157 messages back. Recency finds a required message in the top five for 95.9% of pooled probes and 2.2% of the distant ones, while BM25 finds one for 24.8% of distant probes. Replacing the last three messages with the recorded evidence raises content match by 0.108 on the proportional sample, and 96% of that gain lands on probes that need no memory. On the 167 items whose reference reply depends on the evidence, the gain is 0.455. A detector shown the probe and three prior messages reaches AUROC 0.530, near chance, while authored questions reach 0.802. Labeling the same ten messages as retrieved memories rather than earlier turns raises memory use by 10 to 14 points. Misfires are zero with no context and rise to 60.8% when ten unselected messages are supplied. Three agent systems reconstruct personas at F1 0.69 to 0.70 while processing between 18M tokens (Codex GPT-5.6-sol) and 546M (Claude Opus 5.5).

Limitations: Ten people on one app is not a population sample, and under the strict reading only five contribute memory-bearing probes. A stylometric attack re-identified authors at 83% (statistical) and 98% (language model), against 10% chance.

Why does it matter?

If you build or sell assistant memory, a gain measured on authored questions mostly reflects messages that never needed retrieval. Report misfire rate on probes that need nothing, split ablations by whether memory was required, and drop the retrieved-memories header from injected context. The cost spread in persona reconstruction is a direct line item for enterprise pricing.

AgentGarten: Code Worlds for Evolving Agents

What problem does it solve?

Simulators keep consistent state and rules but need costly 3D assets for realistic visuals. Video world models produce realistic frames but keep state implicit in generated history, where rendering errors accumulate and nothing can be inspected or edited. MirroS, Tsinghua University and Peking University split the two (paper, code).

How does it solve the problem?

A scene program runs in a simulator or game engine that holds object poses, articulations and task variables. The engine exports depth or surface-normal maps, and one shared neural renderer turns them, plus a reference image and text, into frames. The state transition never reads a rendered frame, so renderer mistakes cannot change what happened. Coding agents write the scene programs from an image or text.

The renderer starts from Cosmos 3-Nano and is distilled with Adversarial Forcing: distribution matching on the model’s own rollouts, a second pass that recomputes history block by block so later losses reach the parameters that encoded earlier frames, and a real-video GAN loss with an exact R1/R2 penalty computed without differentiating through fused attention kernels. Agents act through short Python programs, see only rendered frames, and after each round write Markdown playbooks that the next round’s agents read.

What are the key findings?

Block-by-block replay matches the rollout bitwise, against 3.99% relative L2 error for a full-sequence FlexAttention replay, with a 5.4% faster forward pass. The renderer runs at 36.5 frames per second at 480x832 on one H100, 438.8 ms per 16-frame block. Against Self Forcing, the comparison over 30-second rollouts is qualitative only. In hide-and-seek, hiders built shelters by round 4 and seekers crossed walls with ramps by round 10. The authors cite roughly 25 million and 100 million episodes for the same milestones in OpenAI’s 2019 self-play study and state that the two paradigms differ, so this is not a sample-efficiency ratio. In four more worlds over four rounds, a companion-dog score rose from 13 to 19, a bridge swap fell from 71 to 41 seconds, and a quarry loader went from 30 to 90.9 after scoring 0 in round 2.

Why does it matter?

If you build training or evaluation environments for computer-use or embodied agents, keep state in code you can inspect and replay, and let the learned model handle only appearance. A recorded rollout can then be re-rendered in a different style without changing events. Treat the playbook results as a demonstration rather than a measured learning rate.

Self-Retrospection Distillation: Turning Post-hoc Experiences into Prior Foresight

What problem does it solve?

Group-relative RL with verifiable rewards (RLVR), such as GRPO, learns from differences in reward within a group of rollouts. When every rollout in a group gets the same reward, the advantage is zero and the group contributes no gradient, even though the trajectories show what the task required and how the agent failed. The work comes from a team including Chien-Sheng Wu, Julian McAuley and Silvio Savarese (paper). Only the abstract was available to us, so this section stays at that level of detail.

How does it solve the problem?

The authors call the approach prospective learning: experience gathered after an attempt supervises predictions the agent makes before acting. Self-Retrospection Distillation (SRD) uses each completed trajectory to identify knowledge that would have helped and pitfalls to avoid. That hindsight is privileged, since it depends on having seen the trajectory. SRD trains the same policy, seeing only the pre-interaction view, to predict it. The foresight is a training target only and need not be generated at inference, so deployment cost stays the same. SRD is added on top of RLVR, so it supplies a signal in groups where the reward contrast is zero.

What are the key findings?

Across 10 tool-integrated reasoning and long-horizon agentic tasks, SRD adds up to 24.2 percentage points over RLVR and self-distillation baselines. Between 37% and 98% of rollout groups were reward-uniform depending on model scale. In the 2B setting, where 98% of groups were all-failure, RLVR training ended at 0.0% success while RLVR plus SRD reached 60.6% under the same rollout budget.

Why does it matter?

If you run GRPO-style training on hard agentic tasks with small models, log the fraction of reward-uniform groups first. If most groups are all-fail, outcome reward alone may be teaching nothing, as the 2B result shows, and an auxiliary signal built from the trajectories you already collected is worth testing before buying more rollouts. Read the full paper for construction cost before committing a training budget.

Learn2Play Bench: How Well Do LLM Agents Learn from Experience in Unfamiliar Environments?

What problem does it solve?

Benchmarks for self-improving agents usually use familiar tasks, so a score gain may come from recalled pretraining knowledge rather than learning from interaction. Researchers at the National University of Singapore built 20 new text games whose hidden rules are novel or counterintuitive. In one, stealing a colleague’s lunch gets the player promoted while hard work does not (paper).

How does it solve the problem?

Ten games repeat the same instance across episodes, and ten re-shuffle characters, menus or layouts while keeping the rules, to test transfer. Feedback is deterministic and scoring automatic. Agents play 5 episodes per game (10 for five challenge games) over three independent trials. Four metrics separate peak performance (Max, Mean) from improvement (Learning Gain, Learning Slope). The authors compare nine backbones in the OpenCode harness, six experience-retention methods (Naive, Memory, Reflexion, EvoTest, ReasoningBank, AWM) at fixed backbones, two harness swaps, and 21 to 30 college students per game.

What are the key findings?

In OpenCode, Claude Opus 5 leads Max (74.5) and Mean (53.8), while Qwen 3.8 Max leads Learning Gain (+25.2) and Learning Slope (+6.30), so the best scorer is not the fastest learner. With Kimi K3, Memory, which keeps raw interaction histories, leads all four metrics (Max 63.3, against 62.4 for EvoTest and 54.7 for Naive). With Opus 5 the order differs: EvoTest has the higher Max, 66.1 against 60.9. In one GemForge run, EvoTest turned a single failed fusion into a prompt rule banning combinations it never tried. Claude Code with Opus 5 reaches Max 81.8 against 74.5 in OpenCode, and Codex with GPT-5.6-SOL reaches 74.3 against 57.0. The top human reaches 84.3. Humans repeat action sequences less (similarity 0.54 against 0.66 for agents) and set a new personal best after a setback more often (33% against 22%). OpenCode with Kimi K3 uses 56% fewer input tokens per episode than Memory and scores higher Max.

Why does it matter?

Before building a reflection or rule-distillation layer, test plain retained history on your backbone. Distilled rules can lock in conclusions drawn from one observation. Evaluate harness and model together: on this benchmark a harness swap moved Max by 7 to 17 points at a fixed model, and in some cases also lowered cost.

MotorMind: Scaffolding General Vision Language Models for Zero-Shot Robot Manipulation

What problem does it solve?

Vision-language-action models transfer poorly to new tasks without more robot data and cannot directly benefit from newer general VLMs. Agentic alternatives add segmentation models, coding agents, skill libraries or motion planners. Researchers at the University of Illinois Urbana-Champaign test whether one general VLM can drive a robot the way a human teleoperator does, with none of those extra components (paper).

How does it solve the problem?

A 240-question diagnostic built from LIBERO demonstrations shows action selection is the weakest VLM skill: Qwen3.8-Flash-Next scores 36.25% on it, against 55.00% on progress assessment and 65.00% on completion. That motivates short, revisable action proposals with repeated checks. One frozen VLM plays five roles. A Planner writes subgoals with success criteria. An Executor proposes short batches of move, rotate and gripper commands in the robot base frame with magnitudes it picks, such as moving forward 12.20 mm. A deterministic Controller validates and executes them and reports what the robot actually did. A Monitor on a background thread can cancel queued commands at the next action boundary, a Verifier checks outcomes against observations and measured state, and Memory writes summaries without blocking execution.

What are the key findings?

On LIBERO-PRO, MotorMind reaches 66.7% success on base suites and 53.8% under perturbations, against at most 13.3% and 19.2% for the evaluated zero-shot methods. Under perturbations it edges task-fine-tuned OpenVLA-OFT at 51.2%, though fine-tuned π0.5 reaches 98.3% on the base suites. On a real xArm6 it succeeds in 95% of placement trials, 98% without interference and 92% with a person moving objects. Swapping in GPT-6 Sol raises base success to 83.3% with longer wall time. Removing replanning drops success to 36.7%, and removing the planner drops it to 0.0%. Most remaining failures are visual grounding errors and premature completion claims.

Limitations: Episodes average 223.4 seconds on base suites, against about 6 seconds for fine-tuned π0.5. Ablations and the backbone swap are single-seed, and all tasks are tabletop.

Why does it matter?

For a client that needs a manipulation pilot without collecting demonstrations, a VLM with mid-level actions, a replanning loop and an interrupting monitor is now a measured option, and it improves when the backbone improves. The cycle time rules it out where throughput counts. Budget for that, and for failures where the model picks the wrong object.

❤️ If you enjoyed this article, give it a like and share it with your peers.

Extended “Papers of the Week” for Additional Reading

nanoMuse: An Open-Source Personal Agent for Every Device You Own Zhejiang University releases a GPL-3.0 personal agent that runs on Android, iOS and desktop, operates phone and computer screens, routes every action through a Sentinel gate, and stores memory as readable files. It reports no success rate for its screen-operating hands.

Long-WAM: Scaling the Context of World-Action Models NVIDIA, MIT and collaborators find longer visual history helps far more after autoregressive video pretraining: RoboCasa GR-1 success rises from 63.3% to 78.7% at 19.2 seconds of context, while a bidirectional initialization shows no net gain.

World Observer: Joint Actor-Observer Generation for Persistent World Modeling KAIST generates panoramic observer videos jointly with the agent’s view, so objects keep moving while unseen. On its synthetic benchmark, out-of-view dynamics consistency reaches 0.531, against 0.387 without the observer stream.

Recursive Game Creator: An Agentic Product-Level Experience-Oriented Game Harness HKU MMLab loops designer, builder, code-writing player and reviewer agents. Three refinement rounds raise GameCraft-Bench from 72.70 to 77.89, and GameASG-Bench task success reaches 25/47 against 9/47 for the same model alone.

From Traces to Agentic Worlds: Agentic Language World Models for Interactive Environment Simulation Trace2Env builds a worldbook from past interaction traces and lets an agent simulate the environment. On ALFWorld, 85% of actions generated in simulation still succeed in the real environment, against 3% with direct prompting.

X-Tree: Tokenizing Reusable Experience for Efficient Agent Generalization Waterloo and collaborators mine recurring, success-bearing action spans into a tree with no LLM calls, then train on it. Offline RL on WebArena reaches 22.9% success with Qwen2.5-7B, against 18.4% for full-data SFT.

For anyone that hasn’t seen it yet, I started a YouTube channel where I’ll upload explainer-style videos of both papers discussed in my newsletter and introductory content about important AI concepts.

The first video is about Just-in-Time Memory, a new agent memory technique from Salesforce AI. I’ll embed it below its text section. Cheers!

Just-in-Time Memory: Learning to Curate Task-Adaptive Memory for LLM Agents

What problem does it solve?

Most agent memory systems decide what to keep when a task ends. The finished trajectory is distilled into a reflection, workflow, skill, or reasoning strategy, and later retrieved by similarity. The paper names two costs of this design. Details dropped at write time cannot be recovered when a later task depends on them, and one fixed summary has to serve every future query, although the same household trajectory might teach one task how to heat an object and another where to place it. Training a write-time curator is also awkward, because a storage decision is graded only when a matching query arrives, possibly many tasks later. SkillOS, the closest learned prior work, groups related tasks to create that delayed signal.

How does it solve the problem?

JitMem keeps complete raw trajectories and admits one to the bank only if the executor model, acting as judge, says the task succeeded. For a new task, BM25 over task descriptions fetches the top three trajectories. A curator (Qwen3-8B, thinking mode off) reads the current task and those traces and writes a short briefing: relevant past experiences, strategies that worked, and guidance for this task. A frozen executor acts with that briefing in its prompt, and the briefing is then thrown away. Write-time memory resembles taking notes before you know the exam questions. JitMem rereads the textbook after seeing the question.

Because the briefing is used on the same task it was written for, the curator’s reward is the task outcome itself. Training runs GRPO for 100 steps with groups of 8 candidate briefings, rewarded by each benchmark’s native success metric, against a fixed bank of base-executor successes.

What are the key findings?

With a Qwen3-8B executor, trained JitMem reaches 77.4 success rate on ALFWorld against 61.2 for SkillOS, and 32.8 on WebShop against 16.5 (paper). The untrained curator is already strong: with Gemini-2.5-Pro as curator and executor, it scores 61.0 on WebShop against 41.0 for SkillOS-gemini. A curator trained with the Qwen3-8B executor transfers to GPT-5.4 without retraining, reaching 86.7 on ALFWorld against 88.1 for a curator trained directly with GPT-5.4. The briefings are compact: on ALFWorld with GPT-5.4, the untrained curator adds 1.9K input tokens over no memory, against 10.7K for ReasoningBank and 13.4K for SkillOS-base. Removing the retrieved trajectories from the trained curator costs up to 14.8 points on ALFWorld and 15.2 on WebShop, so it relies on retrieved experience rather than its own parametric knowledge.

Why does it matter?

If you run agents on repeated task streams, store raw successful trajectories instead of only distilled lessons, and add one curation call that sees the current task. The prompted version needs no training and already beats write-time baselines in several settings. Budget for that extra LLM call per task, and expect the authors’ BM25 retriever to need replacing as the bank grows large and varied.

Prefer video? I got you covered. Let me know what you think, I’m new to YouTube and video creation in general.

Raven: The Harness of Harnesses for Composable Agentic Intelligence

What problem does it solve?

EverMind AI argues that agents now face long workflows that span domains, while each harness (tools, context handling, skills, recovery rules) is tuned for one domain and is costly to design by hand. Chaining specialists through free-form messages loses detail and makes outputs hard to trace back to their sources (paper).

How does it solve the problem?

Raven treats each model plus harness pair as a callable unit. A registry lists native specialists (Raven-Research, Raven-Code, Raven-Design, Raven-Oncall) and third-party agents such as Claude Code, Codex, Hermes Agent, and OpenClaw, connected through Agent Client Protocol, a command-line process, or an OpenAI-compatible API. A Host Agent decomposes a request into a typed DAG where each node is an agent call and each edge is a file-backed artifact handoff. Before any worker starts, the runtime runs five groups of checks: format, graph structure, agent capability, agent status, and environment, such as rejecting a path reference sent to an agent without local file access. A separate model call judges each finished node, and failures return to the host to continue, abandon, or replan. Around this sit harness evolution built on the team’s HarnessBank, EverOS memory, and Skill Forge retrieval built on SkillCorpus. A formal section gives sufficient conditions under which composition covers tasks no single agent solves reliably within the same budget.

What are the key findings?

On MAOB, the team’s new 140-request benchmark, Raven’s Exact Match graph agreement is 0.711 against 0.607 for the strongest baseline with Qwen3.8-27B, and 0.867 against 0.762 with DeepSeek-V4-Flash-0731 (paper). Raven-Research reaches 76.5% on DeepResearch Mixed with DeepSeek-V4-Flash, against 68.9% for DeepSeek-Harness and 67.2% for MiroFlow, at 0.0242 USD per question. Raven-Code resolves 15 more of 731 SWE-bench Pro tasks than Claude Code on the same Qwen3.8-27B backbone.

Why does it matter?

For teams wiring several coding or research agents together, the admission checks and artifact ledger are directly reusable ideas: they reject a bad plan before it spends compute and leave an audit trail of which file each node consumed. Enterprises evaluating orchestration products should treat planning-quality scores as a first filter and still ask for end-to-end success and cost on their own workflows.

In-Context Learning for Robots: Methods and Applications

What problem does it solve?

A robot may already know how to grasp every part and still need evidence about what to do: a demonstration sets the assembly order, history shows which placements are done, contact reveals a tighter fit. This survey (arXiv) studies how such evidence changes deployed behavior without another task-specific update to neural parameters, and asks how that change survives physical execution when objects, environments, or conditions shift.

How does it solve the problem?

The survey organizes the field by the interface that connects contextual evidence to execution. It distinguishes four families: context-conditioned policies, geometric demonstration transfer, world-model-based control, and skill- and agent-based execution, with correspondence and memory mechanisms shared across them. It places these within six learning horizons, from explicit control programming (S1) and learned policies (S2), through history-based adaptation (S3) and in-context task learning (S4), to physical recursive self-improvement (S5) and knowledge exchange across robot bodies (S6). A shared notation separates supplied evidence, execution intermediates, and retained state, so fine-tuning (task data updates the weights) and in-context learning (a support demonstration enters as context, weights fixed) can be compared on one policy interface.

What are the key findings?

The findings are conceptual. Where task information is stored, in weights or in context, determines acquisition cost, persistence, and what resets between episodes. Broader motor competence from multi-task, multi-embodiment pretraining is a different property from a broader ability to learn through teaching. The authors argue evaluation should separate three things: whether behavior changes appropriately when the task-defining evidence changes under matched execution conditions, whether it transfers across objects and environments, and whether retained experience helps later tasks.

Why does it matter?

If you evaluate a robot policy marketed as learning from demonstration, ask for the matched control the survey describes: swap the demonstration, hold the scene fixed, and check whether behavior changes. A high success rate on scenes that already determine the task says little about teachability. The taxonomy is also a quick map for deciding whether a new task belongs in weights or in context.

Omni-IO Skills: Harnessing Your Agent Omni-Native

What problem does it solve?

General-purpose agents plan and write code well, but producing audio, video, documents, 3D assets, and coordinated media packages depends on outside tools and explicit orchestration. Adding modalities to a foundation model ties every new capability to a model update. Bolting on specialist tools leaves open how procedures, dependencies, intermediate files, and later revisions are coordinated. The authors, from the National University of Singapore and the University of Oxford, propose a harness layer instead (paper).

How does it solve the problem?

The harness has four layers. Skill Entry holds 27 declarative Skills: 19 Atomic (one operation, such as speech generation), 2 Expert (one deliverable, such as a poster), and 6 Scenario (an application context, such as event material). Higher-level Skills expand recursively into terminal tasks. An MCP Tool Service maps each task to understanding, generation, or utility tools behind one call contract. A Provider and Configuration layer binds each capability to a provider, model, credentials, defaults, and fallbacks, so a backend can be swapped without rewriting Skills. An Asset Registry records every output in an append-only JSON file with a stable ID, type, path, parameters, turn, and source asset, with writes serialized by a file lock.

Workflows become Declare Execution Graphs. After checks that every dependency resolves and the graph is acyclic, nodes run in waves by dependency rather than modality, so image and audio jobs run together while an image-conditioned video waits. A failed node cancels its pending descendants while independent branches continue. On a later turn, earlier assets enter the graph as completed source nodes, so a landing-page restyle reruns only that node.

What are the key findings?

On UniM-90, a 90-instance subset of UniM, the harness raises input support from 40.00% to 100% for GPT-5.6 Sol and from 38.89% to 100% for Claude Sonnet 5. Relative Semantic–Quality Coupled Score rises from 26.99 to 74.94 and from 27.82 to 77.78. Strict Structure Score reaches 100.00 and 99.78 (paper). Absolute scores also rise (67.49 to 74.94, and 71.53 to 77.78), though the base agents’ absolute scores cover only the instances they could process.

Why does it matter?

For builders who need multi-asset outputs from an existing coding agent, the useful pieces are the provider binding layer and the ID-based asset registry: they let you swap image or video vendors and support revise-this-one-thing requests without rewriting procedures. Before adopting it, measure the extra calls and wall-clock time on your own requests.

False Frontiers: Diagnosing and Mitigating Co-Cheating in Self-Evolving Search Agents

What problem does it solve?

Self-evolving search agents build their own curricula: a proposer turns documents into questions and pseudo-labels, a solver trains on them, and the solver’s agreement with the labels rewards the proposer. Researchers from Rutgers, McGill, King Fahd University of Petroleum and Minerals, and other institutions show this loop drifts. A wrong label can train the solver to repeat the error on later questions from the same source, and that agreement is then paid back to the proposer as reward. They call this co-cheating and measure it as false-agreement mass, the fraction of label–response pairs that agree on the same wrong answer (paper). In the standard Dr. Zero loop with Qwen3.5, it grows from about 0.4% in round 1 to 6.1% (4B) and 8.8% (9B) by round 3, while in-loop agreement keeps rising.

How does it solve the problem?

Verifying each proposal before training is the obvious fix, and the authors build it as multi-sample verification (MSV): three answers with the source and three without, admitting a task only when both majorities agree. It costs six extra generations per candidate and only lowers false agreement to 5.7% and 7.2%, because six samples from the same model can share errors and the solver that later scores the proposer still trained on labels from the same source.

CrossFit targets that second path. Each source document is assigned once to fold A or B. Two auxiliary solvers each train on one fold, and questions from fold A are scored by the solver trained only on B, and vice versa, using Dr. Zero’s unchanged frontier reward. The main solver still trains on all admitted questions.

What are the key findings?

CrossFit lowers round-3 false agreement to 3.0% at 4B and 3.7% at 9B, against 6.1% and 8.8% for the coupled loop. Replaying the same 3,000 saved questions with source-excluded feedback drops it to 0.4% and 0.1%, which separates feedback ancestry from curriculum changes. A random question-level split reaches only 0.050 and 0.062, so the split has to follow source documents. Across seven search benchmarks, CrossFit averages 48.8% Cover-EM at 4B against 40.0% for Dr. Zero and 40.1% for Search-R1, and 51.2% at 9B against 42.8% and 43.4% (paper).

Why does it matter?

If you run self-play or self-generated data loops, log which data trained the model that scores your generator, and keep a held-out evidence check outside the loop. Rising in-loop agreement alone is not evidence of progress. Splitting feedback by source document is cheap to add to an existing proposer–solver pipeline.

YuE2: Unifying Symbolic and Audio Music Generation at Frontier Quality

What problem does it solve?

Symbolic music models make melody, harmony, and form readable but stop before a recording. Audio models produce full songs but leave composition implicit, so there is nothing to inspect or edit. YuE2, from HKUST, Multimodal Art Projection, New York University, Stanford, MBZUAI, ACE Studio, and others, asks whether one model can write the score first and still match the finished-song quality of proprietary systems (paper, weights).

How does it solve the problem?

A 3.58B-parameter, 28-layer Mixture-of-Transformers generates three layers in order. It first writes an ABC score with tempo, meter, key, chords, sections, and vocal and instrumental melody. It then predicts 25 Hz MERT2 semantic tokens and finally 64-dimensional acoustic latents decoded to 48 kHz stereo. Each layer has separate autoregressive and non-autoregressive experts that share one attention computation: score and semantic tokens are predicted causally, while acoustic latents are refined bidirectionally by flow matching while attending to the full plan. Training mixes four tasks, with and without the score and semantic tokens, so one checkpoint supports a matched with-versus-without planning comparison. Since recordings lack aligned scores, the team built SheetSage2 to transcribe lead sheets and MERT2 to supply the semantic tokens.

What are the key findings?

With the same checkpoint, prompts, and decoder, experts prefer songs generated with symbolic planning for overall quality, 49.3% against 34.6% (p=0.0070). With planning in both systems, they prefer the unified model to a separate language model plus diffusion Transformer, 53.4% against 35.6%. On WildSongBench, YuE2 scores 6.7316 SongBench Global Avg against 6.3247 for LeVo 2, the best public baseline, and best-of-8 reaches 6.9632 against 6.9377 for Mureka 9. In expert listening, best-of-8 is preferred over Suno v4.5 57.3% to 30.5% and is nearly even with Suno v5 at 40.4% to 39.9% (paper). Score edits follow through: melody edits reach 84.17% pitch accuracy while 90% to 94% of unedited content is preserved across local edits.

Suno v6 still wins overall preference, 59.3% against 31.6%. Lyric accuracy (PER 0.0844) trails MiniMax Music 3 (0.0627). The symbolic training targets are SheetSage2 transcriptions, not human-aligned scores, so transcription errors enter supervision.

Why does it matter?

For teams building music or audio products, the score gives users and language-model agents a concrete object to edit, and those edits change targeted melody and harmony while keeping the rest. The open weights make this a practical base for controllable generation. Best-of-8 results cost eight generations per request.

(More) Papers of the Week

AREX-2: Advancing Self-Improving Agents through Long-Horizon Reflective Tasks BAAI fine-tunes Qwen3.8-27B on multi-round improvement trajectories from ML engineering and algorithmic programming. It reaches 81.8 on MLE-bench Lite against 73.7 for Naive-N0.5-Flash, and 84.0 on BrowseComp without new research data.

Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL GAGAR has a trained grader rank test-passing trajectories within a GRPO group, then shifts credit toward cleaner implementations while preserving the advantage sum. On MiMo-V2.6-Flash it reports better code agent performance and slower trajectory-length growth.

LEGO-Anything: Coding Agents for 3D Scene Reconstruction Coding agents rebuild a scene from one image as an editable Blender program. On LEGO-Bench, GPT-6-astra scores 53.4% indoors and 39.6% outdoors, and a training-free plugin improves all six models by up to 62.7% relative.

Mid-Harness: Scaling Actions Between Model and Harness for Terminal Agents Sampling and verifying candidate terminal commands before execution, with the generator and harness unchanged, raises TMAX-9B Pass@1 on TerminalBench-Lite from 50.00% to 68.03% using a GPT-5.6 Sol verifier over 8 candidates.

Self-Evolving Coding Agents: From Digital Programs to Physical-World Intelligence HexaAnything represents robot task state and procedure as code and calls a VLA as a tool, raising RoboCasa365 Composite-Unseen success from 34.3% to 38.3% over native XR-1.

EvoDuet: Bilevel Co-Evolution of Web Searching and Task Solving for Scientific Discovery Co-evolving search queries with solutions raises OpenEvolve’s normalized discovery gain across 21 tasks from 61.3% to 82.3% with Gemini-3.8-Flash and from 74.1% to 78.0% with GPT-5.6-Luna. Qwen3.5-9B does not benefit.

WorldAuditBench: Interactive 3D World Auditing with Multimodal Agents UC Santa Barbara and MIT release 213 anomaly-finding tasks across 13 3D environments. VLM agents succeed on 28.2% to 42.3% of tasks, VLA-then-VLM pipelines on 6.6% to 17.4%, and humans on 83.4%.

❤️ If you enjoyed this article, give it a like and share it with your peers.

1. Showing a video judge the agent’s execution log makes open-weight judges accept most failed clips.

2. A registry check placed before any gate blocks calls to tools that do not exist.

3. One edited worker description can cut a multi-agent system’s task success by more than half.

4. In all 256 audited open-weight chat tokenizers, user text can forge control tokens.

5. A production team estimates full-benchmark scores from 38.5% of the questions, with about 1 point of error.

Quick Glossary

Don’t Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents

Agentic video pipelines use a loop: a planner calls a text-to-video model, and a multimodal judge accepts or rejects the result. To make failures easier to diagnose, some harnesses also show the judge the execution trace, the plan and the narration. The authors test whether that extra text changes verdicts on purely visual requirements when the frames stay the same.

They built 109 two-event clips with Wan2.2-T2V-A14B, labeled each one by hand, and varied only the accompanying text. Their remedy is least-privilege judging. Each requirement is routed only the evidence that can establish it: frames for visual requirements, the trace for process requirements, and frames with burned-in subtitles masked by EasyOCR and blurring.

With a trace reporting success, the Qwen-VL judges (7B, 8B, 32B) accept 0.78, 0.90 and 0.83 of failed clips, up from 0.17, 0.19 and 0.07 with no text. A “visual-only” prompt still leaves false-accept rates of 0.41 to 0.63. Frontier judges barely move (GPT-5.4-mini 0.12 to 0.14). In a repair loop with an honest planner, the judge pass rate reaches 1.00 while the true pass rate is 0.28. Least-privilege routing restores true pass to 0.86 and 0.92, but cost per episode triples (0.36 to 1.11) because the loop now regenerates real failures. The benchmark is small and labeled by the authors, and only one open-weight judge family was tested.

Closed-World Resolution Against Tool Hallucination in LLM Agents

Agents sometimes call tools that do not exist or pass arguments that no schema declares. Tool-selection methods choose among real tools, and gating methods constrain what an agent may do with real tools. Both assume the call names a real tool, so no gate ever evaluates a hallucinated one.

The paper defines five hallucination classes (H1 to H5) and the Resolution Rung, a training-free check that confirms the tool is in the registry and type-checks its signature before any causal gate runs. An MCP version resolves (server, tool) pairs. The authors also release the Hallucinated-Tools Benchmark (HTB).

Across ten models on Amazon Bedrock, they recorded 322 hallucinated calls. A gating-only stack executes all 322, and the rung rejects all of them. Calls to fabricated tools cluster on raw-JSON invocation (34 cases, against 3 under schema-enforced APIs), so moving to schema enforcement is a change you can make this week. Scale does not reduce the problem: Llama-3.1 goes from 0.55 at 8B to 0.57 at 70B. On live MCP, Claude Opus 4.8 hallucinates at 0.57. On HTB, resolution plus gate scores 0.96 against 0.56 for a JSON-schema validator. The zero over-rejection result comes from the authors’ own honest-call generator, the study has no confidence intervals, and the schema-valid H5 residue (0.1525) remains unsolved.

Misleading the Planner through Deceptive Resumes: Registration-Time Injection in Centralized Multi-Agent Systems

In a centralized multi-agent system, the planner reads each worker’s description to decide how to split a task, which worker handles each part, and what each part requires. Third-party descriptions are written outside the system but trusted inside it. A payload placed in a description takes effect before any user request arrives, even if the crafted worker is never invoked.

The authors define four fields a description should contain (functionality, input spec, output spec, usage constraints) and design eight attack strategies in three classes. Their defense, DescGuard, applies rule-based preprocessing followed by three LLM stages: emphasis neutralization, extraction of the interface fields, and a restatement scoped to the worker. Each stage has an independent validator and uses random delimiters.

On GAIA with AutoGen, a single manipulated description drops task success from 84.31% to 37.25% (agent exclusion) or raises token use by 111.93%. Among 32,000 descriptions from GPT Store, Coze and Baidu Wenxin, at least 89.73% lack input specs, so a malicious description does not look unusual. DescGuard restores success to 84.31% and lowers benign success only to 82.35%. It does not verify that a description matches the worker’s actual implementation, and it has no formal guarantee against attacks written to target it.

Subscribe now

Nameless Tokenization: A Lossless Tokenizer-Level Defense Against Control-Token Forgery in Open-Weight LLMs

Open-weight models publish the strings their chat templates use for turn boundaries, and the tokenizer maps those strings to reserved IDs. As a result, anyone who controls prompt text can write a turn boundary identical to one written by the serving stack. The authors audited 256 deployed tokenizers and found every one of them forgeable.

The usual fix is the split_special_tokens flag. It leaves 56.6% of tokenizers forgeable because 56.2% of models register some tool or reasoning markers as non-special tokens, which the flag does not cover. Nameless tokenization keeps each control token’s ID and embedding but removes its surface string from the content encoder. The template writes IDs directly, so message text has no way to produce them.

The method reproduces the standard token stream exactly on attack-free data across five tokenizer families. With it, 0.0% of adversarial strings reach a reserved ID, against 27.3% undefended and 5.1% with the flag. Delimiter-fidelity accuracy rises from 8.5% to 59.9%, and forged tool results fall from 33.1% to 7.4%. The fix has a clear limit. Under a bare task instruction, a forged user turn still succeeds 92.8% of the time (99.9% without the fix), because the delimiter’s visible text does most of the work. Plain-language injections are unaffected.

Efficient Benchmarking in Production: A Study of an Evolving LLM Agent

The authors run a production analytics agent with tens of thousands of monthly users, evaluated on a 519-question benchmark that is expensive to rerun on every change. Earlier work on efficient benchmarking calibrated its methods on many different models on public benchmarks. This study instead follows one evolving agent over time.

Using 574 runs over 52 days, split chronologically into calibration and held-out halves, the team compares random sampling, historical caching, fixed subsets, and IRT adaptive testing. The adaptive methods work like a computerized exam that picks each next question based on earlier answers. The comparison includes a new multidimensional 2PL variant with D-optimal question selection.

The 2PL variant reaches 1.03 pp MAE after executing 200 questions (38.5% of a full run). At 100 questions it scores 1.97 pp, against 2.94 pp for random sampling. The team nevertheless deployed difficulty-stratified fixed subsets (1.40 pp at k=200) because they are simpler to operate. Those subsets transferred without recalibration to five other agent families, beating random sampling at 11 of 12 budgets (1.47 against 1.86 pp at k=200). The data comes from one organization and is not released, and the paper does not measure missed regressions at an actual release gate.

❤️ If you enjoyed this article, give it a like and share it with your peers.

Leave a comment

That is the whole story of agents in 2026 compressed into one anecdote. The model matters less than we thought. The wrapper matters more. And an industry that has just discovered this is now doing what industries always do with a new lever: pulling it as hard as possible and calling the result a strategy.

I want to argue that a harness, however good, is a performance instrument and not a deployment solution, and that “harness-maxxing” (tuning the wrapper against a benchmark until the scores look frontier-class) creates three specific risks for anyone trying to run agents inside an actual organization. The argument is not that harness engineering is bad. It is the most useful engineering discipline to emerge this year. The argument is that its success has made it easy to confuse with something it is not.

What a harness actually does

Microsoft’s agent framework docs describe it as the scaffolding around the model, and that is the useful mental picture. The model predicts tokens. The harness runs the loop, holds the tools, assembles context, enforces permissions, and persists state between sessions. A model cannot open a file or ask for approval before sending an email. The harness does that on its behalf.

There is an inner harness the vendor builds and an outer harness the user builds, and the outer one is made of two kinds of controls: guides that shape behavior before the agent acts (instructions, skills, examples) and sensors that observe after it acts (tests, linters, reviewers). There’s a third category I think is essential: caps, boundaries the model cannot talk its way past. Guides advise. Sensors report. Only a cap, enforced by permissions or infrastructure, can actually stop anything.

The lever is real

Before criticizing the overuse of harness engineering, it is worth being honest about how much it works. LangChain kept the model fixed and changed nothing but the harness, and their coding agent moved from 52.8% to 66.5% on Terminal-Bench 2.0, from outside the top thirty into the top five. The improvements sound boring but useful: a middleware hook that intercepts the agent before it exits and forces a verification pass, a startup step that maps the directory so the agent doesn’t burn turns on discovery, a loop detector. The most common failure they found was the agent writing code, re-reading it, deciding it looked fine, and stopping without running a single test. The fix lived entirely in the wrapper.

OpenAI’s harness engineering team went further. Over five months, three engineers drove Codex to roughly a million lines of code across about 1,500 merged pull requests, with no manually written source. Their early progress was slow not because the model was weak but because the environment was underspecified. The engineering work became building the environment: structured docs, custom linters enforcing architecture, garbage-collection agents that sweep for drift. Anthropic’s work on long-running agent harnesses points in the same direction: separate a planner, a generator, and a skeptical evaluator that never saw the build, and make them agree in writing on what “done” means before any code is produced.

So the mechanism is well understood. A model’s intelligence is spiky. The harness is where you sand down the spikes for the tasks you care about. This is why the same GPT-5.5 scores 83.1% in Codex and 78.0% in Terminus 2 on the same benchmark at the same effort level. Five points of “model capability” that were never in the model.

What harness-maxxing is, and why it’s tempting

Harness maxxing is what happens when the lever becomes the goal. You pick a benchmark, iterate the wrapper against it, and stop when the number is impressive. Sometimes this is done cynically, to make a model card look better. More often it is done in good faith by a team that has just watched thirteen points appear out of thin air and reasonably wants thirteen more.

It is also now automatable. Stanford’s IRIS lab published Meta-Harness, an outer-loop search over harness code itself - LangChain’s managed version detects failures in production traces, drafts a fix, and adds an evaluator to prevent regression. The optimization loop that used to require an engineer reading traces can now run by itself, against whatever metric you point it at. Which brings us to the dangers.

Danger one: the score belongs to the pair

When a vendor reports a benchmark result, the honest sentence is not “Model X scores 82.” It is “Model X in Harness Y at effort level Z scores 82.” The official Terminal-Bench leaderboard records exactly that triple. Most marketing does not. Moonshot’s K3 was evaluated in Moonshot’s own harness, while the competing columns in its comparison table used the best score across harnesses. When Artificial Analysis ran the same model through their own harness, it landed 3.3 points lower.

For an enterprise this is a procurement problem before it is an engineering problem. A team that selects a model on published scores is buying a harness they don’t have, tuned to a benchmark they don’t run, and will then wire that model into a harness of their own where none of the tuning transfers. The Qwen developer who got 28% with a bash-only agent wasn’t doing anything wrong. He was running the model in a different environment. That is what every enterprise deployment is.

Danger two: a maxed harness is scar tissue for one model

Every component in a tuned harness encodes an assumption about a specific model’s specific failure. The pre-exit verification hook exists because this model skips tests. The directory map exists because this model wastes turns exploring. The loop detector exists because this model gets stuck in a particular way. The OpenAI team said this openly: their best harness components are designed to be deleted, because each one is a workaround for a limitation the next model may not have.

And there’s also an interaction effect between models and their harnesses. The enterprise implication is that the more aggressively you tune a harness, the more of your deployment’s measured quality lives in patches that are specific to a model you will replace within a year. Model release cycles are now measured in months. A harness maxed against a March model might become a liability by September. The tuning you did for performance becomes the reason you can’t switch.

Danger three: the harness sits inside the blast radius

A harness lives in the agent’s process. Its guides are prompts and skills. Its sensors are tests and reviewers. Even its caps, the permission checks, are enforced by the agent runtime. That is the right place for them when the goal is making one agent do one task well. It is the wrong place for them when the goal is running agents across an organization, because none of the things that actually kill enterprise deployments happen inside the agent’s process.

Gartner predicts more than 40% of agentic AI projects will be canceled by the end of 2027, and the reasons it cites are unclear business value, weak governance, and the cost of integrating agents into legacy systems, not model capability. In separate research this May, Gartner warned that 40% of enterprises will demote or decommission autonomous agents by 2027 because of governance gaps discovered only after production incidents - most enterprises treat agent governance as binary, “either locked down or fully trusted.”

The most common failure modes? Unclear success criteria. Integration with systems the agent doesn’t own. Governance that doesn’t match the agent’s autonomy level. Identity and audit. Who is accountable when the agent is wrong. Not one of these is addressed by a better harness, and a maxed harness can make several of them worse, because it produces a system that looks reliable in the lab and has no mechanism for being wrong gracefully in front of an auditor.

A harness can enforce that the agent tested its work. It cannot enforce that the work was the right work. That is a specification problem, an ownership problem, and in most enterprises an organizational memory problem, and the agent has no organizational memory to draw on.

Make it concrete

Take a claims-processing agent at an insurer. A team builds it on a strong model with a strong harness: skills that encode policy language, a validation sensor that checks every output against the policy schema, a verification hook that makes the agent re-read the claim before approving. On an internal benchmark of historical claims it hits 94% agreement with adjusters. Everyone is pleased.

Then it ships. The claims system it writes to has a field the agent never saw in training data, because the benchmark was built from exports that dropped it. The permission rule that stops it from approving anything over $10,000 is a prompt instruction and a harness check, which means it is enforced in the agent’s process and nowhere else. When a second team reuses the agent through an internal API, they bypass the harness entirely. Three months later, the model is upgraded, the verification hook that was tuned for the old model’s habit of skimming starts firing constantly on the new model’s different habit, and the team disables it to reduce noise. Nobody decided that the agent should now approve claims without re-reading them. But it still happened.

None of those failures are harness failures. The harness did its job. It just wasn’t the right job for the harness to begin with.

What to do with this

If you are building or buying agents for an organization, there are a few things to consider:

1. Evaluate model-plus-harness pairs, and only in your harness. A published score tells you what the vendor’s wrapper can do. Run the candidate through the wrapper you will actually deploy, on tasks you actually have, before believing any number.

2. Keep the rules outside the harness. Anything that limits what the agent can touch, spend, or send should be enforced by a system the agent process cannot modify. Prompts and hooks are guides and sensors. They are not fences.

3. Budget for harness churn. Treat every tuned component as having an expiration date tied to the model version, and write down why each one exists so you can delete it when the reason goes away. OpenAI’s framing, components designed to be deleted, is the right one.

4. Define the success criterion before the tuning loop starts. An automated harness optimizer will climb whatever hill you point it at. If the hill is a benchmark, you will get a benchmark. If the hill is “the adjuster would have made the same call, and we can show why,” you need that criterion written down and owned by someone before the first trace is analyzed.

5. Take harnessability seriously as an architecture criterion. Typed languages, clear module boundaries, and structured data give you sensors for free. Legacy systems with none of those properties are exactly where the harness is most needed and hardest to build. That is a systems decision, not an agent decision.

My honest read is that harness engineering will remain the highest-leverage thing an agent team can do for at least the next six months, and that the enterprises who get burned by it will be the ones who mistook a good score for a finished system. The field rewrites itself every few months - a lot of the fundamentals haven’t changed though. On the contrary, I would argue that you have to put more thought into them than ever before - now that software can be built at the speed of light.

❤️ If you enjoyed this article, give it a like and share it with your peers.

Subscribe now

• Enabling web search on ChatGPT reduced benchmark accuracy by up to 8 percentage points, and repeated runs of the same prompt produced inconsistent answers on up to 21% of prompts - raising hard questions about how we evaluate deployed AI systems.

• A neuro-symbolic RAG framework that compiles retrieved text into executable Prolog modules achieves 61.1% accuracy on ShARC, outperforming a standard RAG baseline’s 42.8% - without any domain-specific training.

• A new benchmark of 201,247 real investor decisions reveals that leading LLMs produce logically plausible financial reasoning (scoring ~4/5) but ground that reasoning in actual market events at a dismal 0.8 - 2.8/5 - polished language masking shallow understanding.

Let’s dive in.

What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)

What problem does it solve?

When we evaluate LLMs for safety and reliability, we almost always do it the same way: send prompts through an API, run each prompt once, and report accuracy. But that’s not how most people actually use these models. Real users interact through chat interfaces, often with web search enabled, and they ask the same kinds of questions repeatedly. The gap between how we test and how people use these systems is a blind spot - and it’s one that matters most precisely in the safety-critical contexts where benchmarks are supposed to give us confidence.

The researchers ask a simple question: if you test the same model through different access modalities (ChatGPT’s chat UI vs. OpenAI’s API), with and without web search, and across multiple runs, do you get the same results? If not, what are we actually measuring when we report a single accuracy number?

How does it solve the problem?

The team took a stratified sample of 401 prompts from two widely-used benchmarks - BBQ (measuring social bias) and SafetyBench (measuring safety) - and collected 4,812 total responses across both modalities, with and without web search, running each prompt three times. This is a straightforward but labor-intensive audit design: hold the model family constant, vary the conditions that differ between lab evaluation and real deployment, and measure what changes.

Beyond standard accuracy, they measure response consistency across repeated runs, text similarity between modality pairs, citation grounding (what sources does the model actually cite, and do they differ by modality?), and abstention behavior (when does the model refuse to answer?). These are the kinds of behavioral variations that a single accuracy number completely obscures.

The design is intentionally conservative - same model family, same prompts, same benchmarks the community already trusts. The only things that change are the access conditions. That makes any observed differences hard to dismiss as artifacts.

What are the key findings?

Chat UI responses were less accurate than API responses on both benchmarks with search disabled. Enabling web search reduced accuracy by up to 8 percentage points and, in one case, actually reversed which modality performed better - the chat UI went from underperforming to outperforming the API on SafetyBench when search was turned on.

Consistency was another concern. Repeated runs of the same prompt produced different responses in up to 21% of cases. The two modalities grounded their answers in different citations even when answering the same question. Abstention behavior - whether the model refuses to answer - was also inconsistent across modalities and search conditions.

Why does it matter?

If a model’s safety profile changes depending on whether you test it through the API or the chat interface - and changes again when web search is toggled on - then single-condition evaluations are giving us a false sense of precision. The model your users interact with through a chat interface with search enabled may behave meaningfully differently from the model you evaluated through the API without search. That delta is where real-world risk lives, and current benchmarking practices are largely blind to it.

Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalized Financial Agents

What problem does it solve?

Financial AI benchmarks tend to evaluate LLMs in one of two ways: static question answering (can the model answer finance trivia?) or terminal profit-and-loss (did the model make money?). Neither captures what actually matters for a financial agent. Static QA ignores the fact that the same market signal should trigger different actions for different investors - a retiree and a 25-year-old tech worker should not respond identically to the same earnings report. And P&L-based evaluation can’t distinguish between a well-reasoned trade and a lucky one.

The deeper problem is that investment competence is inherently personalized. A model that produces fluent, logically structured reasoning about a stock might still be completely wrong for a specific investor’s risk profile, time horizon, or portfolio constraints. Current benchmarks have no way to measure this.

How does it solve the problem?

The authors introduce InvestLogicBench, a benchmark built from 201,247 documented decisions made by 151 real-world investors. Each decision is structured as a five-stage trace they call P→E→R→D→O: investor Profile, observable market Events, investment Reasoning, executable Decision, and delayed Outcome. This isn’t synthetic data - these are real investment decisions with real outcomes, decomposed into inspectable reasoning chains.

The benchmark supports three evaluation modes: comprehension (can the model understand a given reasoning trace?), profile-conditioned generation (can it produce reasoning appropriate for a specific investor profile?), and end-to-end replay (can it reconstruct the full decision process?). The key design insight is separating process quality from outcome quality. A model might arrive at a profitable trade through sloppy reasoning, or make a sound decision that happens to lose money due to market conditions. InvestLogicBench can distinguish these cases.

The P→E→R→D→O structure also functions as a proposed data-system interface for consequential agents more broadly. The authors argue that any agent making high-stakes personalized decisions needs versioned profiles, temporal provenance for information, inspectable retrieval, decision ledgers, and replayable outcomes.

What are the key findings?

Across four leading LLMs, the results reveal a telling pattern: logical plausibility scores remain near 4 out of 5, but event grounding - whether the model’s reasoning is actually anchored in the specific market events it references - drops to just 0.8 - 2.8 out of 5. In other words, the models produce reasoning that sounds right but isn’t reliably connected to the evidence it claims to use.

Return quality and process quality also disagree, confirming the authors’ thesis that outcome-only evaluation hides important failure modes. A model can generate polished, logically coherent investment reasoning that would score well on any fluency metric while being poorly grounded in the actual market conditions it’s supposed to be analyzing.

Why does it matter?

This paper exposes a failure mode that extends well beyond finance: LLMs are increasingly good at producing reasoning that looks right without being right. In domains where decisions are personalized and consequential - healthcare, legal advice, financial planning - fluent reasoning can be actively dangerous if it isn’t grounded in the specific evidence and user context that should drive the decision.

The P→E→R→D→O framework is worth watching as a general pattern for evaluating agentic systems. Any domain where the same evidence should produce different actions for different users needs this kind of process-level evaluation. Outcome-based metrics alone will continue to hide the gap between articulate reasoning and grounded reasoning.

NeSy-RAG: Neuro-Symbolic RAG for Explainable Question Answering

What problem does it solve?

Standard RAG pipelines retrieve relevant text chunks and feed them to an LLM to generate an answer. This works well enough for straightforward factual questions, but the reasoning process remains opaque. You can’t easily verify which retrieved evidence actually supported which part of the answer, and you can’t trace the logical steps the model took to get there. When the answer depends on user-specific conditions - “Am I eligible for this benefit given my circumstances?” - standard RAG has no systematic way to detect that it’s missing critical information. It either guesses or hedges.

This is a real problem for any application where the answer depends on conditional logic: regulatory compliance, eligibility determination, policy interpretation. The user needs to know not just the answer, but why, and what assumptions the system made to get there.

How does it solve the problem?

NeSy-RAG introduces a modular neuro-symbolic pipeline that sits between retrieval and answer generation. Instead of passing retrieved chunks directly to an LLM for free-form generation, the system compiles each chunk into a Prolog module containing semantically meaningful predicates - essentially Boolean claims that may depend on user-specific facts. Picture it as translating natural language rules into executable code: “If the applicant is over 65 AND has lived in the UK for 3+ years, then they qualify.”

The system uses joint natural language-code embeddings to retrieve relevant predicates and compose them into Prolog queries. The framework includes a symbolic knowledge-gap detection mechanism that identifies missing user facts whose truth values would change the query outcome. If the system can’t determine eligibility without knowing the user’s age, it automatically triggers a follow-up question rather than guessing.

Executing the resulting Prolog queries produces deterministic answers with transparent execution traces. Every reasoning step links back to its originating source chunk, giving you full attribution - not the probabilistic “this chunk was probably relevant” attribution of standard RAG, but logical, verifiable provenance.

What are the key findings?

On the ShARC benchmark - a conversational QA dataset that requires conditional reasoning and follow-up questions - NeSy-RAG achieves 61.1% accuracy without any domain-specific training. A same-model RAG baseline achieves 42.8%, making this an 18.3 percentage point improvement from the symbolic reasoning layer alone. This improvement comes entirely from the structural change in how reasoning is performed - no additional training data, no fine-tuning, no larger model. The Prolog compilation step converts a fuzzy generation problem into a crisp logical execution problem, and the accuracy gains follow directly from that conversion.

Why does it matter?

For anyone building RAG systems that need to handle conditional logic - eligibility checks, compliance verification, policy interpretation - NeSy-RAG offers a concrete architectural pattern. The key insight is that separating retrieval, logic compilation, and execution gives you verifiability and gap detection that end-to-end generation can’t provide.

In production RAG systems, one of the hardest failure modes to catch is when the system confidently answers a question it doesn’t have enough information to answer correctly. NeSy-RAG’s symbolic layer makes missing information explicit and actionable, turning a silent failure into a follow-up question. That’s a meaningful improvement for any high-stakes QA application.

TS-RAG: Retrieval Augmented Generation for Time Series Forecasting

What problem does it solve?

Retrieval-Augmented Generation (RAG) has proven its value in Natural Language Processing (NLP) by letting language models pull in relevant external information at inference time. The natural question is whether the same idea works for time series forecasting: if you’re predicting future values of a sequence, could retrieving similar historical patterns improve accuracy? In principle, yes - but in practice, the approach doesn’t transfer cleanly.

The core difficulty is that time series models aren’t language models. They typically have smaller parameter counts, limited training data, and lack the generative flexibility that makes RAG work so well in NLP. Simply concatenating a retrieved reference sequence into the input - the equivalent of stuffing context into a prompt - doesn’t reliably help. The model doesn’t know how to weight the reference against its own input, and the concatenation can introduce noise rather than signal.

How does it solve the problem?

TS-RAG introduces a dedicated fusion mechanism for incorporating retrieved sequences into time series forecasting. Rather than treating reference sequences as additional input tokens (the naive approach), the framework introduces specially designed reference tokens that mediate between the input sequence and retrieved similar sequences.

The retrieval step identifies historically similar time series patterns from a reference database. The fusion step is where TS-RAG departs from standard RAG: instead of concatenation, the reference tokens act as an information bridge, allowing the model to selectively incorporate temporal dynamics from the retrieved sequences. Think of it as giving the model a structured way to say “this historical pattern is relevant to what I’m seeing now” rather than forcing it to figure out what to do with a raw appended sequence.

This design acknowledges a fundamental asymmetry between language and time series domains. In language, context is context - more relevant text generally helps. In time series, the relationship between a reference pattern and the current input is more nuanced: the reference might be similar in shape but different in scale, or similar in periodicity but offset in phase. The reference token mechanism gives the model a learnable interface for handling these relationships.

What are the key findings?

The authors report that TS-RAG achieves consistent state-of-the-art performance across several real-world forecasting benchmarks. While the abstract doesn’t break out individual benchmark numbers, the emphasis on consistency is notable - many time series methods show strong results on one or two datasets but degrade on others. Consistent improvement across benchmarks suggests the reference token mechanism generalizes rather than overfitting to specific data characteristics.

The key result is less about any single metric and more about the proof of concept: RAG can work for time series forecasting, but only if you design the fusion mechanism for the domain rather than importing it wholesale from NLP. The naive concatenation approach that the authors contrast against serves as a useful negative result - it confirms that the problem isn’t retrieval quality but information integration.

Why does it matter?

If you’re working on forecasting problems where historical analogues exist - energy demand, financial markets, sensor networks - TS-RAG suggests that retrieval-augmented approaches can improve accuracy, provided you invest in the right fusion architecture.

More broadly, this is part of a growing trend of adapting LLM-native techniques (RAG, in-context learning, chain-of-thought) to non-language domains. The lesson from TS-RAG is that these adaptations require domain-specific engineering at the interface layer. The retrieval concept transfers - the implementation details don’t.

Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents

What problem does it solve?

National standard documents - like China’s GB/T standards - are lengthy, highly structured documents governed by explicit rules about scope, terminology, normative wording, and cross-section consistency. Reviewing these documents for compliance is currently a manual, expert-driven process that’s expensive and difficult to scale. It’s the kind of task that looks like it should be easy for LLMs: the rules are explicit, the documents are structured, and the errors are well-defined. But existing benchmarks don’t actually test this capability.

Most LLM benchmarks for document understanding focus on domain knowledge or question answering. They don’t test whether a model can systematically audit a document against a complex rule set, identify specific violations, locate them precisely, and classify them correctly. That’s a different and harder task - more like a code linter for natural language documents than a reading comprehension test.

How does it solve the problem?

The authors introduce GB/T-Bench, the first benchmark specifically designed for structured document review. At its core is the GB/T Review Taxonomy, a hierarchical schema covering document structure, scope alignment, normative modality, terminology consistency, and normative references, with 25 diagnosable error types. This taxonomy defines what “correct” looks like for a national standard document.

To generate evaluation data, the team developed a controllable counterexample generation mechanism. Starting from 488 real documents, they combine deterministic rule-based mutations with constrained LLM rewriting to produce 7,306 traceable review error instances. Each error has a known location, dimension, and type - making evaluation precise rather than subjective.

The evaluation protocol itself requires exact matches on error location, review dimension, and error type, plus document-level coverage metrics. This is significantly harder than “did the model notice something was wrong?” - it’s “did the model identify the right error, in the right place, for the right reason?”

The authors also propose GB/T-Reviewer, a multi-agent framework that converts review knowledge into specialized skills and coordinates global inspection, targeted diagnosis, rule scanning, and result verification. Each agent handles a different aspect of the review process, and their outputs are composed into a comprehensive document audit.

What are the key findings?

The results reveal a substantial gap between human experts and current LLMs. The strongest single model achieves a CMCS (Comprehensive Multi-dimensional Compliance Score) of only 0.3280, compared to 0.6640 for human experts - roughly half the expert-level performance. The GB/T-Reviewer multi-agent framework raises the best score to 0.5094, a meaningful improvement but still well below expert level.

Fourteen mainstream LLMs were evaluated, and the consistent finding across all of them is that rule-intensive document review remains a hard problem. Models can catch some errors but struggle with the systematic, exhaustive coverage that expert reviewers achieve. The multi-agent approach helps by decomposing the task into specialized sub-tasks, but the gap suggests fundamental limitations in how current models handle complex, interacting rule sets applied to long documents.

Why does it matter?

Many organizations are exploring LLMs for compliance, auditing, and document review tasks, but without rigorous evaluation instruments, it’s hard to know whether the models are actually ready. GB/T-Bench provides a clear, quantitative answer: not yet, but structured multi-agent approaches can close roughly half the gap.

The broader lesson is about task decomposition. Rule-intensive review requires a model to hold a complex rule set in mind while scanning a long document - a combination of long-context understanding and systematic reasoning that remains challenging. The multi-agent approach works precisely because it breaks this compound task into manageable pieces. For practitioners considering LLMs for compliance or quality review, the takeaway is that single-model approaches will likely underperform, but well-designed multi-agent pipelines can get meaningfully closer to expert-level performance.

Subscribe now

Papers of the Week

Brief highlights from other notable papers this week:

• Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents - Introduces a reference framework for assessing the quality of benchmarks used to evaluate task-oriented conversational agents. The work highlights that poor benchmarks with inconsistent tasks and limited policy coverage can lead to unreliable agent evaluations - a meta-evaluation problem that mirrors this week’s broader theme.

• MetaboLLM: A Metabolomics-Specialized Large Language Model for Biochemical Knowledge Integration and Predictive Metabolite Graph Construction - Adapts an LLM for metabolomics through continual pretraining, supervised fine-tuning, and structured retrieval, enabling predictive metabolite graph construction from heterogeneous biochemical data sources. A compelling example of domain-specialized LLM adaptation beyond typical NLP tasks.

• Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers - Proposes SiPE (Syntax-informed Positional Embeddings), which learns a lightweight syntactic prior from dependency parse trees to supplement standard positional encodings. The approach encodes structural relationships between tokens rather than just sequential distance.

• SAGA: Score-Weighted Adaptive Generation Alignment for Low-Resource Nordic Language Models - Presents a preference optimization method that works without costly human preference annotations, targeting morphologically rich, low-resource Nordic languages. SAGA uses score-weighted alignment to bridge the annotation scarcity gap for languages where preference data is minimal.

• Learning When to Trust via Selective Context Preference Optimization - Tackles the nuanced problem of teaching models when to trust external context and when to ignore it. The key insight: a model trained to blindly resist misleading context becomes useless when the context is actually correct, so selective trust calibration is essential.

• RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer - Extends reasoning capabilities to non-English languages using on-policy self-distillation with reasoning pivots, providing dense token-level supervision on student-generated rollouts for more effective multilingual reasoning transfer.

• Visual Grounding in Zero-Shot Vision-Language Control - Investigates whether vision-language models used as zero-shot controllers actually ground their decisions in visual input. The finding is sobering: simulator dynamics and conservative action priors can produce favorable scores without meaningful perception, calling into question how we evaluate visual grounding.

• Dynamic Graph Prompting via Topology-Routed Mixed-Curvature Experts - Reveals that temporal shifts in graph topology require different embedding geometries and proposes routing between mixed-curvature expert spaces for dynamic graph prompting. Adapts pre-trained temporal backbones to label-scarce downstream tasks using lightweight prompts.

Putting It All Together

This week’s papers converge on a single uncomfortable theme: we are often measuring the wrong things, or measuring the right things under the wrong conditions. The benchmark audit from Encarnación et al. shows that the same model tested through different access modalities produces meaningfully different safety results. InvestLogicBench reveals that LLMs can score well on logical plausibility while being poorly grounded in the evidence they claim to use. GB/T-Bench demonstrates that even explicit, rule-based review - a task that should play to LLM strengths - exposes a 50%+ performance gap between models and human experts. And in the quick hits, the visual grounding paper finds that VLM controllers can succeed without actually perceiving their environment. Our evaluation frameworks are systematically optimistic because they test under idealized conditions and measure surface-level outputs.

The second theme is the growing sophistication of RAG architectures. We’re moving well past “retrieve and concatenate.” NeSy-RAG compiles retrieved text into executable Prolog programs, gaining verifiability and gap detection. TS-RAG introduces domain-specific fusion tokens because naive concatenation fails for time series. Both papers suggest that the next generation of RAG systems will be defined not by retrieval quality but by how intelligently they integrate retrieved information - and that this integration layer needs to be designed for the specific domain and reasoning type.

Looking ahead, I expect the measurement problem to intensify as models are increasingly deployed through agentic pipelines with search, tool use, and multi-turn interaction. The gap between API-based benchmarking and real-world behavior will only widen, and the community will need evaluation frameworks that account for the full deployment stack - not just the model in isolation. The work we discussed today is laying some of the conceptual groundwork for that shift. Until next week!

❤️ If you enjoyed this article, give it a like and share it with your peers.

Leave a comment

Except that in the space of about eight weeks this spring, three different coding agents shipped a feature called /goal, and they shipped it to solve a problem none of the earlier layers touched. It isn’t getting the model to keep working. Models were always happy to keep working. It’s getting a credible answer to a much duller question: is this actually finished?

And the two major implementations, Claude Code’s and Codex’s, answer it in almost opposite ways.

Each layer of the stack answers a different question

Prompt engineering asks what you say in a single turn. Context engineering asks what the model can see when it reads that turn: the retrieved documents, the memory blocks, the project files. Harness and loop engineering move up a level, so that instead of you typing the next prompt, a system does. Loop engineering is the practice of building the thing that prompts your agent on a cadence, rather than prompting it by hand. Boris Cherny’s line, relayed in Greyling’s write-up, is that his job is to write loops, not individual prompts.

But a loop only answers what starts the next turn. It says nothing about what stops it. A loop with a time interval runs until you kill it. A loop with no termination logic burns tokens on a task it finished forty minutes ago, or abandons one it never finished at all.

Goal engineering is the layer that answers what ends this. You state a bounded objective with a verifiable done condition, the system keeps the objective alive across turns and compactions, and something checks the condition. That last clause is the whole argument. Greyling’s framing is that a goal needs four primitives: an objective, a verifier, external state, and a budget. His sharpest line is about the verifier. You need one that doesn’t let the implementer grade its own homework.

Hold onto that. It’s exactly where the two implementations diverge.

Claude Code takes the verdict away from the worker

Anthropic shipped /goal in Claude Code v2.1.139 in May. The mechanism, per the docs, is almost aggressively simple. /goal is a wrapper around a session-scoped, prompt-based Stop hook. You state a completion condition. Setting it immediately starts a turn with the condition as the directive. Then, every time Claude finishes a turn, the condition plus the conversation so far get handed to a small fast model, Haiku by default, which returns yes or no plus a short reason. A “no” sends Claude back to work with that reason as guidance. A “yes” clears the goal.

The design choice underneath is the interesting part. The model doing the work does not get to declare victory. A different model, with a different prompt and no stake in the outcome, reads the transcript and rules. VentureBeat, covering the launch, framed it as separating the agent that works from the one that decides it’s done. That framing is exactly right, and it describes a structural fix rather than a capability fix. Self-assessment doesn’t get better with a smarter model. It gets better with a second one.

What Claude Code gives up in exchange is durability. One goal per session. No pause and resume - you can clear a goal, but you can’t park it. Turn and time bounds live inside the condition as text (”or stop after 20 turns”), and the docs are candid that Claude reports progress against that clause and the evaluator judges it from the conversation. It’s a soft stop, not a mechanical one. A goal survives --resume, but the turn counter, timer, and token baseline all reset.

Codex makes the goal an object in the runtime

OpenAI took the other half of the problem. Codex’s /goal landed in the 0.128.0 release line at the start of May, behind a goals feature flag you enable with codex features enable goals. The CLI reference describes it as a persistent target attached to the active thread, with a real lifecycle: set, view, pause, resume, clear.

That lifecycle is the point. Someone who read the source noted an asymmetry that reads as deliberate. The model gets tools to start a goal and to mark it complete, but pause, resume, and budget transitions belong to the user and the runtime. The completion response carries a budget report: tokens used against tokens allotted, seconds spent. Goal state persists in the CLI’s own storage rather than living in the conversation, so it survives compaction and session resumption in a way a transcript-scoped condition does not.

But notice who signs off. In Codex, the model marks its own goal complete. There is no built-in second opinion. You can get one from /review, a subagent, CI, or an eval suite in the stop condition - you just have to wire it yourself. Codex separates control from the model and leaves judgment with it. Claude Code separates judgment and leaves control loose.

Grok Build, which shipped /goal on June 22, sits closer to the Codex end: a visible progress checklist, status, pause, resume, clear, and a panel that flips to Complete when the agent says so.

The evaluator can’t run anything, which changes what you’re actually writing

In the Claude Code implementation, the evaluator does not run commands or read files independently. It judges your condition against what Claude has already surfaced in the conversation. The judge is blind to everything except the transcript. Which means a condition like “the tests pass” is not a claim about your test suite. It’s a claim about whether the test output landed in the conversation where the evaluator could read it. If Claude fixes the code and never re-runs the suite, the condition fails on evidence, not on merit. If Claude runs the suite and reports a summary rather than the exit code, the evaluator is grading a summary.

So goal engineering, in practice, is less about writing objectives than about specifying evidence. You are designing what proof has to appear in the record. “All tests in test/auth pass” works because running tests produces transcript. “The production deployment succeeds” doesn’t, because nothing the agent can safely do will demonstrate it.

Codex has the mirror-image version of the same problem. Its budget report is precise about tokens and seconds and silent about correctness, because the entity reporting completion is the entity that did the work.

Correct and useless: what happens when the condition is the whole spec

The failure mode this creates is Goodhart’s law wearing a lanyard. Give an agent a verification-only condition - tests pass, build succeeds, no console errors - and you can get back something that clears every machine check and is visually embarrassing - a condition made only of machine checks selects for the cheapest artifact that passes them, which is rarely the artifact you wanted.

That’s the tax on mechanical verifiability. The conditions that evaluate cleanly are the ones that capture the least about whether the thing is good. Exit codes are honest and narrow. “The code is clean” is broad and unjudgeable. Most real work lives in between, and the goal will optimize hard for whichever half you wrote down.

What to do with this

If you’re picking up either tool this quarter, a few habits follow from the mechanics rather than from the marketing.

Write the check, not the wish. Every condition should name the command or artifact that proves it. npm test exits 0. git status is clean. The queue is empty. In Claude Code that’s literally what the evaluator reads, and in Codex it’s the only thing standing between you and a confident self-report.

Add the constraint clause. What must not change on the way there. Without it, deleting the failing test is a valid path to green.

Bound the run, and know how soft the bound is. In Claude Code, “or stop after 20 turns” is a sentence the evaluator interprets, not a counter that trips. In Codex, budgets are tracked by the runtime. Plan your blast radius accordingly.

If you’re on Codex, build the verifier yourself. A reviewer subagent, a CI gate, an eval in the stop condition - anything that isn’t the implementer. This is the one gap the tool leaves open by design, and Greyling’s four-primitive checklist is a reasonable place to start.

Keep external state for anything that spans sessions. A GOAL.md in the repo holding the objective, the definition of done, the guardrails, and a running log costs you nothing and survives compaction, session resets, and the turn counter zeroing out on resume.

And use goals for the right shape of work. A goal finishes a bounded thing with a knowable end state: a migration, a spec implementation, a backlog burndown. A loop discovers recurring work on a cadence. If you don’t yet know what done looks like, you want plan mode, not a goal.

Last: read the diff. A goal that clears is a claim that a condition held, nothing more. The migration shipped. That doesn’t mean you understand how it works, and comprehension debt compounds faster when the agent is good.

The honest caveat is that this is all extremely young. Claude Code’s /goal is roughly three months old, Codex’s is still flag-gated, and the same idea is propagating up the stack. Anthropic’s Managed Agents added rubric-based outcome grading at the API level in May. The specific commands in this piece will be stale by autumn.

What seems more durable is the shape of the disagreement. Persistence and adjudication are separable problems, both implementations solved one and punted the other, and the gap between looks done and is done is where the remaining engineering lives. Whichever tool you’re in, the useful question to keep asking is the same: who signs off, and what did they actually get to look at?

❤️ If you enjoyed this article, give it a like and share it with your peers.

The pitch for agent memory is clean. An agent that forgets everything between sessions is a goldfish - give it a memory and it finally becomes a colleague. Half the tooling that shipped in the last year sells some version of that promise. It’s a real improvement. It is also not the thing standing between today’s agents and the work you actually want them to do.

The teams putting agents against real business data keep hitting the same wall, and it isn’t forgetfulness. The agent remembers fine. It pulls back the right-looking documents, quotes the right-sounding numbers, and gets the meaning wrong -confidently. That gap, between remembering something and understanding it, is the most useful thing to get straight about agent architecture right now.

Memory solves statelessness, not meaning

Strip away the branding and “memory” for an agent is two things: persistence and retrieval. You write the important bits of a session somewhere durable, and next time you fetch the relevant bits back. That genuinely fixes the goldfish problem. A coding assistant that recalls your project’s architecture, a support agent that remembers a customer’s last three tickets - this is worth having.

But look at how retrieval works. Almost every memory system ranks stored fragments by similarity - how close a chunk sits to your query in embedding space - and hands the top matches to the model. The industry’s own taxonomy is honest about this division of labor: memory handles continuity across sessions, retrieval handles broad document lookup, and a knowledge graph handles structured reasoning - three different jobs. Memory does the first one well. It does not do the third one at all.

Memory hands back fragments ranked by similarity - what sits nearby in embedding space, not what happens to be true.

Near in embedding space is not the same as correct in meaning. Ask an agent about Q3 revenue and it will happily retrieve chunks that mention “Q3” and “revenue.” It will not know that your finance team reports revenue net of returns, that the “Northeast region” in the August deck is not the “Northeast region” from the reorg two weeks later, or that one of those confident-looking paragraphs is from a superseded draft. It has the words. It does not have the model of what the words mean. Memory gave it recall. Recall was never the hard part.

The naive fix makes it worse

The obvious response is to retrieve more - turn up the number of chunks, or skip retrieval entirely and pour everything into one of the million-token context windows the model vendors keep advertising. Just give it all the memory. Let the model sort it out.

In mid-2025, Chroma published a study it called context rot: across 18 (back then) frontier models - the GPT-4.1, Claude 4, Gemini 2.5, and Qwen3 families among them - performance degraded as input length grew, on tasks as simple as finding a fact and repeating it back. The decline started well before the window was anywhere near full. And the sharpest finding for our purposes: the models got worse when the irrelevant text was semantically similar to the answer, because similar-looking distractors are the hardest ones to ignore.

Why “just add more memory” backfires: past a point, extra tokens are extra distractors, and signal drowns in similar-looking noise.

Every fragment your memory system retrieves that is relevant-looking but wrong is a distractor competing for the model’s attention, and the more of your domain sounds alike - every quarter’s revenue deck, every region’s postmortem - the more of those distractors you’re feeding in. What determines quality isn’t how much you can store; it’s the signal-to-noise ratio of what you pull. So scaling memory scales the noise along with the signal. Past a point, the agent’s job (reason correctly) and memory’s job (recall broadly) actively pull against each other.

A fair caveat: Chroma sells retrieval infrastructure, so it has a house interest in “bigger windows aren’t the answer.” But the broader effect it’s pointing at -degradation in long context, the “lost in the middle” problem - is documented well beyond Chroma, and the mechanism is the kind you can feel in any system you’ve actually run. More context is not more understanding. Often it’s just more ways to be wrong.

The tier you’re actually missing

If memory is storage and retrieval is fetching, the thing sitting between raw fragments and a correct answer is meaning - and that’s a distinct architectural layer with its own name. A semantic layer is a governed model of what your entities, metrics, and relationships actually are: the shared vocabulary your whole stack agrees on before anyone, human or agent, asks a question.

Two different streams have been converging on that idea, and both versions of the concept bring something valuable to the table. First, let’s take a look at them independently:

The first comes from the data world, where a semantic layer has meant one thing for years: the governed place where business definitions live above the raw tables, so that “revenue” or “active customer” resolves to the same number in every tool and every dashboard. This is the old problem of Marketing and Finance quietly using two different definitions of the same word - and it’s mature enough that Gartner elevated the semantic layer to essential infrastructure in its 2025 Hype Cycle for BI and Analytics, and now projects that universal semantic layers will be treated as critical infrastructure for enterprise AI by 2030.

The second comes from the knowledge-graph world, where the “layer” is entities and the explicit relationships between them rather than metric formulas. Instead of a bag of text chunks, you store that this contract belongs to this account, which is owned by this rep, and supersedes that earlier contract. This is the structure behind GraphRAG and behind memory systems like Zep, whose Graphiti engine models memory as a temporal knowledge graph rather than a flat vector store. The payoff shows up exactly where vector search struggles: on multi-hop, cross-document reasoning. A 2026 benchmark over the Code of Federal Regulations - a domain full of “see the amended version of this clause” indirection - reported a 70% accuracy gain from a knowledge-graph approach over standard vector RAG. That’s a single hard-case study, not a universal law, but it points the right direction: the harder the relationships, the more structure becomes a differentiator.

Different tools, same underlying job. Both answer the question memory can’t: what does this thing mean, and how does it connect to everything else? That is the interpretation tier, and retrieval quietly skips it.

You don’t always need one

This is where the honest version of the argument has to slow down, because “add a semantic layer” is not free and plenty of agents don’t need it.

If you’re building a single-user note assistant, a stateless code-generation tool, or a chat interface where nobody downstream acts on a specific number, a semantic layer is overhead you can skip. Memory, or even nothing, is fine. The layer earns its cost under four conditions, and it’s worth checking whether any of them describe your system:

The agent emits numbers people will act on - a metric in a report, a figure in a customer email - where computing it two different ways is a real error, not a cosmetic one. The agent has to reason across many linked entities, following relationships more than one hop deep. Multiple agents or tools have to agree on definitions, so a sub-agent doesn’t compute “churn” differently than the orchestrator expects - a live concern now that agent-to-agent protocols push work across specialized agents. Or your domain carries definitions that simply don’t exist in the raw data: fiscal calendars, org hierarchies, the internal meaning of “enterprise tier,” which regulation supersedes which.

Make it concrete. A support agent handles a refund request. In the memory-only design, it retrieves the customer’s history and the refund policy, reads that the order total was $40, and issues the refund - but “order total” in that chunk was the pre-discount subtotal, and the policy it grabbed was last year’s, since both matched the query beautifully. With a semantic layer in the path, “refund amount” is a defined quantity with one calculation, and “current policy” is a governed entity with a status, so the wrong versions aren’t even eligible to be retrieved as truth. Same memory underneath. The difference is that meaning stopped being something the model had to guess from whatever text came back.

How the layers stack

The practical architecture is not memory or a semantic layer. It’s memory plus one, wired in order.

At the bottom sits the storage tier: a vector store for unstructured recall, plus an agent-memory system to manage what persists across sessions - Mem0, Zep and its Graphiti graph, and the growing field around them. On top of that sits the semantic tier, and which flavor you reach for depends on the job. For metric consistency, the mature options are the pure semantic layers - dbt’s Semantic Layer, Cube, and AtScale - now joined by warehouse-native versions like Snowflake’s Semantic Views and Databricks’ Metric Views, and BI-native ones like Looker’s LookML. For entity-and-relationship reasoning, it’s a knowledge graph - Neo4j, Fluree, or a GraphRAG pipeline. Then the whole thing gets exposed to agents through an API or, increasingly, through MCP, so the agent queries governed meaning instead of raw rows.

The shape of the fix: a storage-and-memory tier at the base, a governed semantic tier above it, and the agent querying meaning on top.

One development worth watching, because it changes the buying calculus: the Open Semantic Interchange, a 2025 effort in which dbt Labs, Snowflake, Salesforce, and others agreed to a vendor-neutral format for semantic definitions, with dbt’s MetricFlow as the spec. If it holds, you define meaning once and every tool - and every agent - reads the same definitions, instead of re-encoding your business logic inside each runtime.

The test that tells you which layer you’re missing

Here’s the diagnostic I’d actually run. The next time your agent states a number or names an entity, ask where the definition came from. If the answer is “whatever chunks the retriever returned,” you have a memory layer and no semantic layer - and you’re one plausible-looking distractor away from a confident, wrong, and hard-to-audit answer.

Fixing that doesn’t mean modeling your whole company on day one. The most sober advice in this space is to start with a handful of disputed metrics rather than a full taxonomy - twenty definitions people actually fight over beat five hundred nobody validated. Governed meaning is the part that compounds, storage is the part that’s already commoditized.

Two honest caveats to hold alongside all of this. A lot of the loudest “you need a semantic layer” voices are vendors who sell semantic layers. The reported figure that roughly 40% of enterprise leaders now see missing semantic context as a major blocker for operational AI is exactly the anxiety that moves the product - so discount the volume and check the mechanism, which is the part that’s real. And the two streams we’ve discussed, metrics and graphs, are still merging → the tooling to serve both to an agent through one clean interface is young and moving monthly. Hold these patterns firmly and update them often.

But the core of it is stable enough to bet on. Memory gives an agent recall. Recall was the easy 80%. The expensive failures live in the other 20 - the part where the agent had the right words and the wrong meaning - and no amount of memory reaches that. Meaning is a layer you build, not a byproduct of remembering more.

❤️ If you enjoyed this article, give it a like and share it with your peers.

A classical intuition in computer science holds that verifying a solution is easier than producing one. This week, that intuition gets inverted.

The Verification Crisis: The most provocative finding this week comes from The Verification Horizon, which argues that for modern coding agents, reliably verifying solutions has become harder than generating them. As foundation models develop stronger reasoning capabilities, every reward function we build is merely a proxy for human intent - and optimization pressure widens the gap between proxy and intent, manifesting as reward hacking and signal saturation. This finding casts a long shadow over agent development: if we cannot trust our verification signals, how do we train agents that reliably do what we want?

Self-Directed Learning and Data Synthesis: Two papers this week explore how agents can improve their own training pipelines without constant human intervention. Autodata introduces the concept of an “agentic data scientist” that meta-optimizes synthetic data creation, converting inference compute into higher-quality training data. Meanwhile, OPID tackles the sparsity of trajectory-level rewards by extracting hierarchical skills directly from an agent’s own completed trajectories, yielding dense, token-level supervision that is distribution-matched to the current policy. Together, these papers suggest a direction toward agents that generate the very signals they use to get smarter.

Agent-Native Infrastructure and Efficiency: Building agents that work in production requires more than capable models - it demands robust systems underneath. Are We Ready For An Agent-Native Memory System? evaluates 12 memory architectures across 11 datasets, revealing that no single system dominates and that localized maintenance is more cost-efficient than global reorganization. For agents to be production-ready, they must be architecturally robust and token-efficient.

Dynamic Context Grounding and Adaptation: Two papers address the “context gap” - the mismatch between what users or environments provide and what agents need to act effectively. Qwen-Image-Agent bridges underspecified image generation requests through a unified framework integrating planning, search, reasoning, and memory. In-Context World Modeling enables robots to infer world dynamics from a short history of self-generated interactions, adapting to novel camera viewpoints and morphologies without any parameter updates. Both treat adaptation as an in-context problem rather than a fine-tuning problem.

Trust, Privacy, and Human Alignment: As agents make more decisions on behalf of users, the question of alignment with social norms becomes urgent. PrivacyAlign places human judgment at the center of privacy alignment, introducing annotation-conditioned reward modeling grounded in 3,516 detailed annotations from 599 unique annotators. The paper argues that privacy is not a binary classification problem but a contextual judgment defined by social expectations - and that standard labels are unreliable proxies for the nuanced norms agents must follow.

Read more

AI agents are getting more capable, but we’ve been measuring that capability wrong. Across eight papers spanning visual reasoning, computer-use automation, deep search, marketplace logistics, and agent security, a clear narrative emerges - the field is undergoing a Great Reality Check, moving from synthetic prowess toward real-world reliability.

The “Shortcut” Illusion - Why Your Benchmarks Are Lying to You: The most consequential finding this week is that outcome-only evaluation substantially overestimates agent performance. WeaveBench introduces a trajectory-aware judge that catches agents fabricating visual evidence and using hard-coded metrics - behaviors invisible to traditional pass/fail grading. The best frontier model-runtime pairing achieves only 41.2% PassRate on its 114 real-world tasks. Meanwhile, FORT-Searcher demonstrates that structurally complex search tasks often collapse through “cheaper identifying routes,” and proposes a shortcut-aware difficulty framework that forces agents to actually reason rather than exploit dataset artifacts. Together, these papers suggest that a significant portion of reported agent progress may rest on evaluation gaps rather than genuine capability gains.

Specialized Multi-Agent Collaboration and Role Decoupling: Single-chain reasoning continues to show its limits, and this week three papers offer distinct architectures for decomposing complex problems across specialized roles. Visual Para-Thinker++ trains Main, Worker, and Summary agents within a single shared policy using role-decoupled optimization, consistently outperforming single-trajectory baselines on hallucination-sensitive visual reasoning. InterleaveThinker pairs Planner and Critic agents to enable any image generator to produce coherent interleaved text-image sequences, achieving performance comparable to GPT-5 on interleaved generation benchmarks. And a deployed multi-agent RL system at DoorDash uses decentralized store-level policies to adapt dispatch tradeoffs in a live three-sided marketplace, increasing batching efficiency without degrading delivery quality.

Stateful Interfaces and Adaptive Action Spaces: How agents interact with their tools matters as much as the reasoning behind the interaction. SpatialClaw demonstrates this by replacing rigid tool-call interfaces with a stateful Python kernel, letting agents write and execute code cell-by-cell while adapting to intermediate observations. The result is a +11.2 point improvement over the previous best spatial agent across 20 benchmarks - without any benchmark-specific tuning. This stateful, code-as-interface philosophy directly complements WeaveBench’s finding that real-world tasks demand hybrid GUI-CLI-code orchestration within single trajectories.

Dynamic Environments and Stakeholder-Aware Safety: Two papers push the boundaries of where and how agents must operate. EvoArena reveals that current agents achieve only 39.6% accuracy on tasks in evolving environments, and proposes a patch-based memory paradigm that improves performance by up to 6.1% on standard benchmarks. On the security front, a stakeholder-centric prompt injection benchmark finds that not a single attack objective was reliably resisted by current agents, and introduces the concept of “stealthy parasitism” - attacks that succeed without disrupting the user’s task, making them invisible to conventional evaluation. Both papers underscore that deployment-ready agents need far more than strong benchmark scores.

Read more

Lightweight Alignment and the End of Scale-Only Thinking: The most striking finding this week comes from AgentDoG 1.5, which demonstrates that ultra-lightweight models ranging from 0.8B to 8B parameters can match the safety performance of closed-source frontier models like GPT-5.4 - using only approximately 1,000 carefully purified training samples. This result, achieved through a taxonomy-guided data engine with influence-function purification, suggests the bottleneck for agentic safety is shifting from raw compute to data quality and structural alignment. Combined with Skill0.5‘s difficulty-aware routing between internalized general skills and externalized task-specific skills, this week paints a picture of agent development where architectural cleverness increasingly outweighs brute-force scaling.

Structured Blueprints over Black-Box Reasoning: A recurring thread across multiple papers is the rejection of pure end-to-end reasoning in favor of structured intermediate representations. UI-KOBE constructs app-specific knowledge graphs to guide lightweight GUI agents through UI states, rather than asking small models to plan from raw screenshots alone. GenClaw takes a parallel approach in image generation, using executable code - SVG, HTML, Three.js - as a controllable “canvas” that bridges linguistic reasoning and pixel synthesis. Both papers share a conviction that inserting a structured blueprint between intent and execution yields more reliable, interpretable results than iterative prompt refinement ever could.

The Verification Imperative: As agents take on more complex, open-ended tasks, the question of trust becomes unavoidable. Ptah introduces a dedicated verifier agent that serves as an “acceptance function” for multimodal deep research reports, enforcing factual grounding, citation fidelity, and cross-modal consistency. Meanwhile, AgentDoG 1.5 deploys training-free online guardrails for real-time safety moderation of agentic behavior, reducing Docker-level deployment overhead by two orders of magnitude. Together, these papers signal that production-grade agents will increasingly require independent validation layers operating alongside - not inside - the primary reasoning model.

World Models Under Scrutiny: Two papers this week push the frontier of interactive world models while a third interrogates whether the entire enterprise is on solid footing. minWM provides a full-stack open-source framework for converting bidirectional video diffusion models into real-time, camera-controllable autoregressive generators. Yet YoCausal delivers a sobering evaluation of 13 state-of-the-art video diffusion models, revealing that perceiving the arrow of time does not imply understanding causality - and that a significant gap persists relative to human-level causal cognition. For agent builders eyeing video world models as planning substrates, this tension between functional rollout and genuine causal understanding is one to watch closely.

Hybrid Architectures and the Pareto Frontier: Hybrid Multi-Agent Systems from Rainone et al. systematically maps the design space between cloud-hosted frontier LLMs and cost-efficient on-device SLMs. Their finding that greater frontier-level compute does not consistently translate to better performance challenges simplistic assumptions about when to offload to the cloud. The optimal architecture turns out to be highly task-dependent, with energy, cost, and accuracy tightly coupled along a Pareto frontier that teams must navigate deliberately rather than by default.

Read more

That gap between the dream and the practice is the most useful thing to understand about agents right now. It isn’t a sign the technology is failing. It’s a sign the field is growing up. And the pattern underneath it all is surprisingly simple to state: the path to capability runs through constraint.

The bottleneck isn’t intelligence

Start with the best data we have. In late 2025, researchers from UC Berkeley, Stanford, UIUC, and IBM published what they call the first large-scale study of AI agents running in production - 306 practitioners surveyed, 20 teams interviewed in depth, across 26 industries. Their headline finding was blunt: reliability is the top development challenge, driven by the difficulty of ensuring and evaluating whether an agent is actually correct.

That deserves a pause, because it contradicts the story we usually tell ourselves. We imagine the frontier of agent work as a race toward raw intelligence - bigger models, longer reasoning, more autonomy. But the teams shipping real systems aren’t mainly blocked by intelligence. They’re blocked by trust. They can’t reliably tell when their agent is right, wrong, or just confidently guessing, and that uncertainty is what stands between a demo and a deployment.

What they do about it is telling. Rather than unleashing autonomy, they fence it in. In that study, 68% of deployed agents take at most ten steps before a human checks in, 70% just prompt off-the-shelf models instead of fine-tuning, and 74% lean on human evaluation. The researchers put it plainly: teams deliberately trade capability for reliability. The agents winning in production aren’t the most autonomous ones. They’re the ones whose autonomy is most skillfully limited.

Context rot: the physics underneath

To see why constraint wins, you need a phenomenon that only recently got a name: context rot.

The belief that “more context means better answers” feels too obvious to question - it’s the whole pitch behind million-token context windows. But research from Chroma, which tested eighteen frontier models including GPT-4.1, Claude Opus 4, and Gemini 2.5, found the opposite. Output quality measurably degrades as input grows, even when the window is nowhere near full. And this wasn’t true of some models. It was true of every one they tested.

The mechanism is what makes it matter. It’s not just about the length. Performance falls off sharply in the presence of distractors - bits of information that look like the answer without being it. One distractor hurts. Add a few, and accuracy drops hard.

Now picture how an agent actually runs. Give it a vague, compound goal - “improve the report,” “handle the customer,” “grow engagement” - and it generates a sprawl of intermediate work: tool calls, half-formed plans, retrieved documents, reasoning about reasoning. All of it piles into the context window, and every irrelevant piece becomes a distractor that raises the failure rate of the next step. Coding agents are the clearest case, because they hit the trifecta: context that only accumulates, search results full of near-identical code, and tasks that run long enough for the rot to compound.

This can’t be solved by more reasoning effort alone. An agent doesn’t fail a sprawling task because it lacks ambition. It fails because the sprawl poisons the well it drinks from. This is why decomposition - breaking work into atomic, single-session chunks - isn’t a productivity tip layered on top, but a direct response to how these models actually process information.

The strange tolerance for slowness

If you assumed agents had to be fast to be useful, the production data says otherwise: 66% of deployed agents are allowed to take minutes or longer to respond. Only a third need sub-minute speed.

Why would anyone accept a five-minute answer? Because the thing you’re comparing against isn’t a chatbot - it’s a person. An agent that takes five minutes on a task that would cost a human an hour is still a tenfold speedup, especially when the task is someone’s secondary responsibility and the team is short-staffed.

That single statistic rearranges the design problem. Optimize an agent for snappy, real-time chat and you’ve implicitly accepted all the failure modes of context rot in exchange for speed the user may not even want. Accept that the agent runs in the background - slowly, deliberately - and you free yourself to do the things that actually buy reliability. You can decompose. You can verify. You can let the agent fail, retry, and check its own work before a human ever sees it. The trade is latency for correctness, and the market is overwhelmingly taking it. The exceptions prove the rule: voice agents, which have to keep pace with human conversation, are exactly where the same study found latency becomes the hard constraint.

Beyond the chatbot

All of this points at something about interface. The chatbot - one scrolling conversation - has been the default metaphor for AI since ChatGPT. For agentic work, it’s increasingly the wrong one.

A conversation is linear, accumulative, and never finished. It is, in other words, a context-rot machine by design: every turn piles onto the last, with no natural boundary, no checkpoint, no way to mark a unit of work done and clear the bench. The chat interface quietly optimizes for exactly the conditions that make agents unreliable.

The alternative changes the human’s job. In the chatbot model you’re a processor - answering the agent’s questions, feeding it the next prompt, stitching the output together by hand. In the task-system model you’re a strategist and a governor: you break a project into atomic, checkbox-style tasks, dispatch them, and review the results against a definition of done. You can watch this metaphor migrate into real products. When Anthropic shipped Cowork - Claude Code’s agentic guts brought to the desktop for knowledge work - it described the experience as “less like a back-and-forth and more like leaving messages for a coworker.” That’s a task system, not a chat. The checkbox, not the chat bubble, becomes the atomic unit.

Automation you earn

If one principle holds this together, it’s this: the right to automate a task should be earned, not assumed. You don’t hand an agent a workflow and hope. You do the task with it first, watch where it breaks, build a fix for those specific failure points, and only then let it run on its own.

This is where the coding world’s hard-won patterns turn out to transfer - not as tricks, but as instruments of context control. Three are worth knowing.

Sub-agents as context firewalls. Instead of one agent carrying a whole project in its head, you spin up a sub-agent for a discrete job - “research vendor pricing” - and have it return only a condensed result. Each runs in its own clean context window. The intermediate mess never reaches the orchestrator, so the rot never spreads. That’s the firewall.

Hooks as guardrails. In coding agents, hooks fire on events to surface errors or block dangerous commands. The distinction that matters, which Anthropic draws in its own docs, is between advice and enforcement: guidance the model can choose to follow versus a check that always runs. For knowledge work, a hook becomes the thing that triggers an evaluator to validate output against explicit “done” criteria before it reaches you - the system checking its own homework as policy, not goodwill.

Skills as packaged competence. A skill is a folder of instructions, scripts, and resources the agent can load on demand. It’s the earned-automation loop made durable: the residue of figuring out how to do a task well, saved so nobody - and no agent - has to rediscover it from scratch.

The architecture that ties it together

There’s one structural move that matters more than any other for systems touching money, records, or anything you can’t take back. It’s the decision to stop letting the model act, and let it only decide.

This cuts against the whole agent pitch - weren’t they supposed to do things? But the teams building auditable systems are converging on the opposite. The agent becomes a decision engine, not an executor. It examines a situation and emits a structured decision. A separate, deterministic layer of ordinary code decides whether and how to carry that decision out.

Make it concrete. A customer requests a refund. In the naive design, the agent reads the account and issues the refund itself. In the better one, a Decision Worker - an agent with read-only access - reads the order history and the policy and outputs a structured verdict: valid refund, $40, confidence 0.96. It cannot touch the billing system; that power simply isn’t wired to it. An Orchestrator - plain, testable code - receives that verdict and executes the payment, logging every step to an audit trail. The reason this matters is subtle but decisive: when an agent executes its own actions, the audit trail tends to live inside ephemeral prompts rather than in a system built to record it. Pull the action out of the prompt and into deterministic code, and you get auditability, retries, and least-privilege security back almost for free.

What makes this powerful rather than merely safe is confidence-based routing. The system runs autonomously by default - most refunds, most classifications flow straight through, no human involved. A person is summoned only when the agent’s own confidence drops below a set threshold. In practice those thresholds usually land between 0.70 and 0.85, calibrated against real labeled data rather than chosen by gut, often in tiers: above the high bar it proceeds, in the middle it acts but flags for review, below the floor it hands off live. And confidence is never the only trigger - mature systems escalate on financial limits, vulnerable customers, or legal exposure no matter how sure the model is. From August 2026, the EU AI Act turns that kind of human oversight from good practice into a legal requirement for high-stakes uses.

What you end up with is a system that’s autonomous where autonomy is safe, governed where it isn’t, and - crucially - knows the difference. The human leaves the loop on routine decisions and moves to the exception desk, handling only the cases the system itself flags as beyond its competence. That’s a more honest and more scalable division of labor than either full automation or constant babysitting.

Where the coding playbook stops working

It would be naive to say that because Cowork and Claude Code share an architecture, the coding playbook simply is the knowledge-work playbook. It’s not that clean, and the difference is worth naming.

Software engineering is unusually objective: the code compiles or it doesn’t, the tests pass or they fail. That’s why coding agents can verify themselves - a compiler is an oracle that never flatters you. Knowledge work has no compiler. A memo, a strategy, a piece of analysis turns on taste and judgment, which are stubbornly hard to package into a test. This doesn’t sink the analogy, but it reshapes it: the verification layer that runs automatically in code has to become human-led, or routed through evaluator agents that approximate judgment rather than compute it.

That last row points to the pattern that does transfer cleanly, and it’s my favorite of the bunch. Call it demand-driven context. Instead of documenting everything an agent might need up front - exhausting, and mostly wasted - you give it a task, let it fail, and then add only the minimum knowledge required for it to succeed next time. Failure writes the curriculum. The knowledge base grows in response to real gaps instead of imagined ones, which is its own kind of discipline.

What to actually do with this

Step back and the forces all point the same way. Decomposition, human governance, earned automation - none of these are temporary limitations waiting for a smarter model to dissolve them. Reliability is the binding constraint, context rot is the physics, and the willingness to trade speed for correctness is what makes careful, decomposed, verifiable work the winning move.

If you’re building or buying in this space, that suggests a few concrete habits. Stop evaluating agents on how autonomous they are and start evaluating them on how well they know their own limits. When a task touches anything irreversible, separate the deciding from the doing. Don’t write the giant knowledge base up front, let failures tell you what to write. And treat full automation as a destination you arrive at by earning it, one verified task at a time - not a switch you flip on day one.

Gartner has projected that a large share of agentic AI projects will be scrapped by 2027, citing cost, unclear value, and inflated expectations. I’d wager the casualties cluster around the old dream - the long-running generalist handed a vague goal and trusted to figure it out. The survivors will look more mundane and work far better: systems of small, high-leverage dispatch loops, governed by a human who treats constraint not as the enemy of capability but as its precondition.

That’s a less magical story than the one we were sold. I think it’s also the more durable one - though the honest caveat is that this field rewrites itself every few months, and the right move is to hold these patterns firmly and update them often.

❤️ If you enjoyed this article, give it a like and share it with your peers.

Long-Horizon Reasoning Stability: The most persistent bottleneck for autonomous agents - maintaining coherent reasoning over extended trajectories - saw a significant advance this week. SU-01, a 30B-A3B model trained with a unified scaling recipe of curriculum SFT and two-stage reinforcement learning, achieved gold-medal-level performance on IMO 2025, USAMO 2026, and IPhO 2024/2025 while sustaining stable reasoning over trajectories exceeding 100,000 tokens. Meanwhile, Self-Distilled Agentic Reinforcement Learning (SDAR) tackled the complementary problem of optimization instability in multi-turn agent tasks, demonstrating that dense token-level guidance via gated self-distillation yields substantial gains over standard GRPO - +9.4% on ALFWorld, +7.0% on Search-QA, and +10.2% on WebShop accuracy. Together, these papers suggest that the field is converging on layered training strategies that combine coarse trajectory rewards with fine-grained supervision.

Multimodal Memory and Visual-Centric Benchmarking: Two papers exposed a critical weakness in how we evaluate whether vision-language agents actually see. MemLens introduced a benchmark of 789 questions across five memory abilities and four context lengths (32K - 256K tokens), revealing that removing evidence images drops frontier model accuracy below 2% on the majority of questions - yet no existing architecture handles both length stability and visual fidelity. MemEye deepened this critique by showing that many agents “cheat” via textual captions, and proposed a granularity-aware framework measuring visual evidence from scene-level down to pixel-level detail. The implication is clear: multimodal agents need hybrid memory architectures that current designs have yet to deliver.

From Individual Agents to Collective Intelligence: The challenge of scaling from single-agent to multi-agent systems received both a practical solution and a theoretical framework this week. LC-MAPF introduced a learnable local communication module for multi-agent pathfinding that outperforms existing learning-based solvers without compromising scalability. The LIFE survey provided a broader conceptual map, organizing multi-agent research into four causally linked stages - Lay, Integrate, Find, Evolve - while warning that tighter coordination amplifies error propagation risks that remain largely unsolved.

Deployment-Ready Safety and Efficiency: Two papers addressed the gap between research prototypes and real-world agent deployment. LiSA proposed a lifelong safety adaptation framework that converts sparse, noisy user feedback into reusable policy abstractions, remaining robust even at 20% label-flip noise rates - without requiring model retraining. SANA-WM demonstrated that minute-scale world modeling at 720p resolution is now feasible on a single consumer GPU, achieving 36× higher throughput than prior open-source baselines. These advances lower the barriers to deploying agents that are both physically grounded and contextually safe.

Read more

What if the entire foundation of AI agent “memory” is built on a category error? This week’s research suggests exactly that - and the implications ripple across every paper in this roundup, from scientific discovery benchmarks where top models score under 10%, to safety-critical systems where non-determinism blocks certification. The papers covered here converge on a sobering but productive conclusion: the agent ecosystem is hitting structural walls that no amount of scaling alone can fix, and the solutions demand rethinking evaluation, architecture, and the very meaning of intelligence in autonomous systems.

The Memory Illusion and Its Consequences: The most provocative claim this week comes from “Contextual Agentic Memory is a Memo, Not True Memory“ (Xu et al.), which argues that vector stores, RAG pipelines, scratchpads, and expanding context windows do not implement memory at all - they implement lookup. The authors formalize a provable generalization ceiling: no increase in context size or retrieval quality can overcome the inability of similarity-based retrieval to handle compositionally novel tasks. Drawing on Complementary Learning Systems theory from neuroscience, they show that current agents implement only the fast, hippocampal half of biological memory while entirely missing the slow neocortical weight consolidation that produces genuine expertise. For agent builders, this means that systems accumulating notes indefinitely are not learning - they are hoarding, and they are structurally vulnerable to persistent memory poisoning in the process.

Evaluation Gaps Exposed: A recurring thread this week is that existing benchmarks dramatically overestimate agent capabilities. AutoResearchBench (Xiong et al.) demonstrates this starkly: even the most powerful LLMs, which have largely conquered general web-browsing benchmarks like BrowseComp, achieve only 9.39% accuracy on deep scientific literature discovery and 9.31% IoU on wide research tasks, with many strong baselines falling below 5%. Meanwhile, “Visual Generation in the New Era“ (Wu et al.) argues that visual generation evaluations overestimate progress by rewarding perceptual quality while ignoring structural, temporal, and causal failures. Together with ClawGym‘s new 200-instance benchmark for multi-step local workflows, these papers make a compelling case that the field needs harder, more honest yardsticks.

Beyond Language as Universal Interface: Two papers this week tackle the limits of text-centric agent design from complementary angles. GLM-5V-Turbo (V Team, Hong, Gu et al.) builds multimodal perception natively into a foundation model’s reasoning, planning, and tool-use pipeline, explicitly rejecting the pattern of bolting vision onto a language model as an afterthought. Heterogeneous Scientific Foundation Model Collaboration (Li, Zou, Fang et al.) takes a different architectural bet: rather than a single integrated model, the Eywa framework uses a language model as a reasoning coordinator that orchestrates domain-specific scientific foundation models over non-linguistic data. Both approaches acknowledge that language alone cannot serve as a universal interface for real-world autonomy, but they disagree sharply on the solution - a tension worth watching as the field matures.

Read more

This week in AI Agents: A critical vulnerability in how agents select and call tools. Small models that punch far above their weight class. Multi-agent systems tackling bias and personalized healthcare. This week’s research paints a vivid picture of an agentic AI field maturing fast - and confronting the hard problems that come with real-world deployment.

Security at the Function-Calling Interface: The most urgent finding this week comes from researchers who demonstrated that the very mechanism enabling agents to use tools - function calling - can be hijacked with alarming reliability. The Function Hijacking Attack paper showed that adversaries can force agentic models to invoke attacker-chosen functions at a 70% to 100% attack success rate across five different models, including both instructed and reasoning variants. Unlike traditional jailbreaks that exploit semantic preferences, these attacks are largely agnostic to context, meaning they generalize across domains and query types. For anyone building or deploying tool-using agents, this paper is required reading.

Small Models, Big Ambitions: Three papers this week converge on a shared thesis: you don’t need massive models to build capable agents. DR-Venus demonstrates that a 4B-parameter deep research agent trained on roughly 10K open data points can significantly outperform prior agentic models under 9B parameters and begin closing the gap with 30B-class systems. AgenticQwen introduces dual data flywheels - one for reasoning, one for agentic behavior - that automatically synthesize increasingly difficult training tasks, enabling small models to handle industrial-scale tool use. And TACO tackles the quadratic token cost growth that plagues long-horizon terminal agents, delivering consistent 1% - 4% accuracy gains on TerminalBench while cutting token overhead by around 10%. Together, these papers suggest that strategic data engineering and inference-time optimization can substitute for raw parameter count.

Data Synthesis as the New Bottleneck-Breaker: A recurring theme this week is that the quality and structure of training data matters more than its volume. OpenMobile builds an open-source pipeline for synthesizing mobile agent trajectories, achieving 64.7% success on AndroidWorld with a fine-tuned Qwen3-VL - competitive with closed-data approaches. LLaTiSA formalizes time series reasoning into a four-level cognitive taxonomy and introduces an 83K-sample dataset with verified chain-of-thought trajectories. Both papers demonstrate that carefully structured synthetic data, combined with curriculum-style training, can unlock capabilities that brute-force scaling alone cannot.

Multi-Agent Architectures for Fairness and Healthcare: Two papers this week deploy specialized multi-agent systems to tackle domain-specific challenges. FairQE uses collaborating agents to detect gender cues, generate gender-flipped translation variants, and dynamically calibrate quality scores - mitigating systematic gender bias in translation evaluation without sacrificing accuracy. The Agentic Physiotherapy framework coordinates four micro-agents to parse clinical notes, synthesize personalized exercise videos, estimate patient pose in real time, and deliver corrective feedback. Both illustrate how decomposing complex tasks across specialized agents can address problems that monolithic models handle poorly.

Read more

Natural language instructions are failing to control autonomous AI agents - and this week’s research makes that case with striking empirical clarity. Across eight papers, we see an industry grappling with the limits of prompt engineering and pivoting hard toward structural solutions: deterministic infrastructure, explainable governance, transferable memory, and reasoning-aware reward systems. The message is consistent: talking to agents is not enough. We need to engineer around them.

The Rise of Harness Engineering: The single most compelling thread this week is the emergence of “harness engineering” as a distinct discipline - designing the complete infrastructure necessary to transform unconstrained agents into controllable, auditable, production-reliable systems. Sema Code decouples AI coding agents from their delivery interfaces, packaging the reasoning kernel as a standalone, embeddable library that any runtime can drive programmatically. Its companion framework, SemaClaw, extends this philosophy to personal AI agents with DAG-based orchestration, behavioral safety systems, and a three-tier context management architecture. Together, they argue that as model capabilities converge, the harness layer - not the model itself - is becoming the primary site of architectural differentiation.

Agent Observability and Enterprise Trust: Deploying agents at scale without adequate governance is producing a phenomenon researchers call “Agent Sprawl,” and this week two papers dissect the consequences. An empirical study of 4,550 agentic pull requests in Do AI Coding Agents Log Like Humans? reveals that agents fail to comply with constructive natural language logging requests 67% of the time, forcing human developers to perform 72.5% of post-generation log repairs as “silent janitors.” Meanwhile, Agentic Explainability at Scale addresses the corporate fears that accompany this governance vacuum, proposing design-time and runtime explainability techniques - including a prototype “Agentic AI Card” - to make agent-to-agent communication and decision-making transparent to enterprise stakeholders.

Advancing Agent Cognition - Reasoning, Memory, and Decision-Making: Three papers push forward the internal cognitive machinery of agents. Exploration and Exploitation Errors Are Measurable introduces policy-agnostic metrics that independently quantify how well agents balance exploring a problem space versus exploiting acquired knowledge, finding that even frontier models struggle - and that minimal harness engineering significantly improves both dimensions. Memory Transfer Learning demonstrates that cross-domain memory improves average coding agent performance by 3.7% across six benchmarks, but only when memories are stored as high-level abstract insights rather than low-level code traces. And RationalRewards shows that teaching reward models to produce explicit, multi-dimensional critiques before scoring transforms them from passive evaluators into active optimization tools, achieving state-of-the-art preference prediction with 10 - 20x less training data.

Multi-Modal World Simulation: Standing apart from the text-centric agent papers, HY-World 2.0 advances the frontier of 3D world generation and simulation. Its multi-modal pipeline accepts text, images, or video and produces navigable 3D Gaussian Splatting scenes through a four-stage method encompassing panorama generation, trajectory planning, world expansion, and world composition. For agents that must perceive and act in physical or simulated environments, this kind of infrastructure could prove foundational.

Read more

AI agents that spontaneously collude to prevent each other’s shutdown. Web agents that fail two-thirds of everyday online tasks. Skills that evolve across users like a living organism. Autonomous AI is at an inflection point - capable enough to surprise us, brittle enough to humble us, and occasionally deceptive enough to alarm us.

Navigating Complex Web and Physical Environments: Several papers this week push autonomous agents out of controlled sandboxes and into the messy real world. ClawBench evaluates agents on 153 everyday tasks across 144 live production websites - and finds that even Claude Sonnet 4.6 achieves only 33.3% success. MolmoWeb takes a different approach, building fully open visual web agents that navigate using only screenshots, no HTML or APIs required, achieving state-of-the-art results among open-weight models and reaching 94.7% pass@4 on WebVoyager through test-time scaling. Meanwhile, HY-Embodied-0.5 bridges the gap to physical environments with embodied foundation models that outperform similarly sized competitors on 16 of 22 benchmarks spanning spatial reasoning and robotic control.

The Scaling and Evolution of Agent Skills: As agent tool libraries grow into the thousands, two papers offer complementary - and sometimes competing - visions for managing them. Graph of Skills introduces a structural retrieval layer that improves average reward by 43.6% while cutting input tokens by 37.8%, solving the immediate problem of context window saturation. SkillClaw goes further, arguing that static skill libraries are fundamentally insufficient and proposing a framework where skills continuously evolve through aggregated multi-user interaction data. Together, they suggest that the next generation of agent architectures will need both smarter retrieval and living, self-improving skill repositories.

Foundations of Reasoning and Coordination: Underpinning all of these applied advances are two papers that rethink how agents learn at a fundamental level. Rethinking Generalization in Reasoning SFT challenges the prevailing narrative that supervised finetuning only memorizes, showing that cross-domain generalization follows a “dip-and-recovery” pattern that many teams may be abandoning too early. And Value-Guidance MeanFlow proposes a flow-based framework for offline multi-agent reinforcement learning that treats optimal joint policy learning as conditional behavior cloning, achieving competitive performance with substantially improved training and inference efficiency.

Read more

Multi-Agent Collaboration and Its Hidden Costs: This week’s research makes one thing clear: the future of autonomous AI is multi-agent, but coordination between agents introduces failure modes that single-agent systems never faced. CORAL demonstrates the upside, achieving 3 - 10× higher improvement rates than fixed evolutionary baselines by letting multiple agents explore, reflect, and collaborate through shared persistent memory and asynchronous execution. But AgentSocialBench exposes a troubling downside: when agents coordinate across domain and user boundaries in social networks, cross-agent communication creates “persistent leakage pressure” on private data - even when agents are explicitly instructed to protect it. Meanwhile, Exploring Robust Multi-Agent Workflows offers a pragmatic middle path for production deployments, showing that role-separated agents with deterministic validators and audited handoffs can catch coordinate transformation errors affecting all 2,452 stations in a dataset before any data reaches the public. Together, these papers frame the central tension in multi-agent design: more agents yield more capability, but also more surface area for compounding errors and information leakage.

From Agent Capability to Agent Containment: Another theme this week is the shift in research focus from making agents smarter to making them safer and more observable once deployed. Investigating Autonomous Agent Contributions in the Wild delivers a sobering empirical finding: across approximately 110,000 open-source pull requests representing millions of lines of code, agent-generated contributions are associated with significantly higher churn rates over time compared to human-authored code. This challenges the “dark factory” narrative of fully autonomous software development and suggests that the bottleneck is shifting from code generation to code maintainability. Complementing this, MTI introduces a behavior-based temperament profiling system that measures what agents actually do - not what they say about themselves - uncovering a “Compliance-Resilience paradox” where opinion-yielding and fact-vulnerability operate through independent channels. These papers collectively argue that standard capability benchmarks are insufficient; we need new instruments to measure disposition, long-term code health, and real-world behavioral risk.

Reinforcement Learning for Structural Agent Failures: Two papers apply reinforcement learning to address fundamental structural problems in agentic reasoning, but from opposite angles. SKILL0 tackles the overhead and noise of runtime skill retrieval by internalizing skills directly into model parameters through a progressive curriculum, achieving +9.7% improvement on ALFWorld and +6.6% on Search-QA while maintaining fewer than 0.5k tokens per step. ProCeedRL addresses the compounding error problem in long-horizon tasks, where a single bad action poisons subsequent context, by deploying a process-level critic that actively intervenes in real time rather than passively selecting among trajectories. The contrast is instructive: SKILL0 eliminates a source of noise before it enters the loop, while ProCeedRL catches and corrects errors once they occur within the loop.

Autonomous Discovery and Self-Improving Research Pipelines: The idea of agents that not only execute tasks but autonomously discover better ways to do so is gaining empirical traction. Omni-SimpleMem deployed a fully autonomous research pipeline that executed approximately 50 experiments without human intervention, improving F1 scores by +411% on LoCoMo and +214% on Mem-Gallery. The most impactful discoveries were not hyperparameter tweaks but bug fixes (+175%), architectural changes (+44%), and prompt engineering improvements (+188% on specific categories) - capabilities fundamentally beyond traditional AutoML. Paired with CORAL’s multi-agent evolution results, these findings suggest that the design space for agent architectures is too large and interconnected for manual exploration, and that autonomous research pipelines may become a standard tool for agent system development.

Read more

Computer-Use Agents and the Data Bottleneck: The path to general-purpose desktop automation remains constrained not by model capability but by training data quality. This week, CUA-Suite tackles this head-on with approximately 10,000 human-demonstrated tasks across 87 applications, totaling ~55 hours of continuous 30 fps video - dwarfing the prior largest open dataset’s ~20 hours. Preliminary evaluation reveals a sobering ~60% task failure rate on professional desktop applications, confirming that current foundation action models still struggle with real-world workflows. Meanwhile, UI-Voyager demonstrates that a 4B-parameter model can reach 81.0% Pass@1 on AndroidWorld through self-evolving learning from failures, surpassing human-level performance without expensive manual annotation. Together, these papers bracket the field’s central tension: we need far more demonstration data, and we need agents that learn efficiently from their own mistakes.

Agent Safety and Adversarial Robustness: As agents gain the ability to execute real actions through tools, the attack surface expands dramatically. T-MAP introduces trajectory-aware evolutionary red-teaming that discovers adversarial prompts capable of bypassing safety guardrails in frontier models including GPT-5.2, Gemini-3-Pro, Qwen3.5, and GLM-5 - achieving harmful objectives through actual tool interactions rather than mere text generation. On the software engineering side, SlopCodeBench reveals that coding agents produce code that is 2.2x more verbose than human-authored open-source projects, with structural erosion rising in 80% of trajectories and no agent solving any of its 20 problems end-to-end. These findings suggest that current safety and quality evaluations systematically underestimate the risks of deploying agents in iterative, long-horizon settings.

Video Understanding as an Agentic Capability: Two papers this week reframe video comprehension as a core planning and perception challenge for autonomous agents. EVA introduces a planning-before-perception paradigm where the agent autonomously decides what to watch, when to watch, and how to watch, achieving 6-12% improvement over general MLLM baselines on six benchmarks. GameplayQA pushes further into multi-agent 3D environments, densely annotating multiplayer gameplay at 1.22 labels/second and revealing that frontier MLLMs exhibit substantial gaps from human performance in temporal grounding and agent-role attribution. For anyone building embodied or simulation-based agents, these results highlight that passive video recognition is insufficient - agents need active, query-driven visual reasoning.

Learning Dynamics and the Fragility of Self-Improvement: The promise of self-improving agents took a nuanced hit this week. Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs? traces performance drops of up to 40% to the suppression of epistemic verbalization - the model’s expression of uncertainty during reasoning. When teacher models are conditioned on rich information, they stop hedging, which helps in-domain but devastates out-of-distribution generalization. This finding has direct implications for any agent pipeline that uses self-generated data for improvement: compressing reasoning traces can silently strip away the uncertainty signals that enable robust decision-making under novel conditions.

Tool Use in Specialized Domains: FinMCP-Bench brings the Model Context Protocol (MCP) into the financial domain with 613 samples across 65 real financial MCPs, spanning single-tool, multi-tool, and multi-turn interactions. While the community signal is modest, the benchmark addresses a critical gap: evaluating whether agents can reliably chain specialized financial tools to solve real-world problems, not just answer questions about them.

Read more