Welcome to this issue of LLM Watch. The six pieces this week cover how to secure and check agents in production. They look at where agent code runs, which inputs judges and planners should trust, and how cheaply you can re-test an agent that changes every week.
Showing a video judge the agent’s execution log makes open-weight judges accept most failed clips.
A registry check placed before any gate blocks calls to tools that do not exist.
One edited worker description can cut a multi-agent system’s task success by more than half.
In all 256 audited open-weight chat tokenizers, user text can forge control tokens.
A production team estimates full-benchmark scores from 38.5% of the questions, with about 1 point of error.
Quick Glossary
Control token: a reserved token ID that marks turns, roles or tool results in a chat template.
MCP: Model Context Protocol, a standard for exposing tools to agents from multiple servers.
IRT: Item Response Theory, a statistical model that estimates question difficulty and test-taker ability from pass/fail outcomes.
Don’t Read the Log: Execution Traces Contaminate Verifiers in Video-Generation Agents
Agentic video pipelines use a loop: a planner calls a text-to-video model, and a multimodal judge accepts or rejects the result. To make failures easier to diagnose, some harnesses also show the judge the execution trace, the plan and the narration. The authors test whether that extra text changes verdicts on purely visual requirements when the frames stay the same.
They built 109 two-event clips with Wan2.2-T2V-A14B, labeled each one by hand, and varied only the accompanying text. Their remedy is least-privilege judging. Each requirement is routed only the evidence that can establish it: frames for visual requirements, the trace for process requirements, and frames with burned-in subtitles masked by EasyOCR and blurring.
With a trace reporting success, the Qwen-VL judges (7B, 8B, 32B) accept 0.78, 0.90 and 0.83 of failed clips, up from 0.17, 0.19 and 0.07 with no text. A “visual-only” prompt still leaves false-accept rates of 0.41 to 0.63. Frontier judges barely move (GPT-5.4-mini 0.12 to 0.14). In a repair loop with an honest planner, the judge pass rate reaches 1.00 while the true pass rate is 0.28. Least-privilege routing restores true pass to 0.86 and 0.92, but cost per episode triples (0.36 to 1.11) because the loop now regenerates real failures. The benchmark is small and labeled by the authors, and only one open-weight judge family was tested.
Closed-World Resolution Against Tool Hallucination in LLM Agents
Agents sometimes call tools that do not exist or pass arguments that no schema declares. Tool-selection methods choose among real tools, and gating methods constrain what an agent may do with real tools. Both assume the call names a real tool, so no gate ever evaluates a hallucinated one.
The paper defines five hallucination classes (H1 to H5) and the Resolution Rung, a training-free check that confirms the tool is in the registry and type-checks its signature before any causal gate runs. An MCP version resolves (server, tool) pairs. The authors also release the Hallucinated-Tools Benchmark (HTB).
Across ten models on Amazon Bedrock, they recorded 322 hallucinated calls. A gating-only stack executes all 322, and the rung rejects all of them. Calls to fabricated tools cluster on raw-JSON invocation (34 cases, against 3 under schema-enforced APIs), so moving to schema enforcement is a change you can make this week. Scale does not reduce the problem: Llama-3.1 goes from 0.55 at 8B to 0.57 at 70B. On live MCP, Claude Opus 4.8 hallucinates at 0.57. On HTB, resolution plus gate scores 0.96 against 0.56 for a JSON-schema validator. The zero over-rejection result comes from the authors’ own honest-call generator, the study has no confidence intervals, and the schema-valid H5 residue (0.1525) remains unsolved.
Misleading the Planner through Deceptive Resumes: Registration-Time Injection in Centralized Multi-Agent Systems
In a centralized multi-agent system, the planner reads each worker’s description to decide how to split a task, which worker handles each part, and what each part requires. Third-party descriptions are written outside the system but trusted inside it. A payload placed in a description takes effect before any user request arrives, even if the crafted worker is never invoked.
The authors define four fields a description should contain (functionality, input spec, output spec, usage constraints) and design eight attack strategies in three classes. Their defense, DescGuard, applies rule-based preprocessing followed by three LLM stages: emphasis neutralization, extraction of the interface fields, and a restatement scoped to the worker. Each stage has an independent validator and uses random delimiters.
On GAIA with AutoGen, a single manipulated description drops task success from 84.31% to 37.25% (agent exclusion) or raises token use by 111.93%. Among 32,000 descriptions from GPT Store, Coze and Baidu Wenxin, at least 89.73% lack input specs, so a malicious description does not look unusual. DescGuard restores success to 84.31% and lowers benign success only to 82.35%. It does not verify that a description matches the worker’s actual implementation, and it has no formal guarantee against attacks written to target it.
Nameless Tokenization: A Lossless Tokenizer-Level Defense Against Control-Token Forgery in Open-Weight LLMs
Open-weight models publish the strings their chat templates use for turn boundaries, and the tokenizer maps those strings to reserved IDs. As a result, anyone who controls prompt text can write a turn boundary identical to one written by the serving stack. The authors audited 256 deployed tokenizers and found every one of them forgeable.
The usual fix is the split_special_tokens flag. It leaves 56.6% of tokenizers forgeable because 56.2% of models register some tool or reasoning markers as non-special tokens, which the flag does not cover. Nameless tokenization keeps each control token’s ID and embedding but removes its surface string from the content encoder. The template writes IDs directly, so message text has no way to produce them.
The method reproduces the standard token stream exactly on attack-free data across five tokenizer families. With it, 0.0% of adversarial strings reach a reserved ID, against 27.3% undefended and 5.1% with the flag. Delimiter-fidelity accuracy rises from 8.5% to 59.9%, and forged tool results fall from 33.1% to 7.4%. The fix has a clear limit. Under a bare task instruction, a forged user turn still succeeds 92.8% of the time (99.9% without the fix), because the delimiter’s visible text does most of the work. Plain-language injections are unaffected.
Efficient Benchmarking in Production: A Study of an Evolving LLM Agent
The authors run a production analytics agent with tens of thousands of monthly users, evaluated on a 519-question benchmark that is expensive to rerun on every change. Earlier work on efficient benchmarking calibrated its methods on many different models on public benchmarks. This study instead follows one evolving agent over time.
Using 574 runs over 52 days, split chronologically into calibration and held-out halves, the team compares random sampling, historical caching, fixed subsets, and IRT adaptive testing. The adaptive methods work like a computerized exam that picks each next question based on earlier answers. The comparison includes a new multidimensional 2PL variant with D-optimal question selection.
The 2PL variant reaches 1.03 pp MAE after executing 200 questions (38.5% of a full run). At 100 questions it scores 1.97 pp, against 2.94 pp for random sampling. The team nevertheless deployed difficulty-stratified fixed subsets (1.40 pp at k=200) because they are simpler to operate. Those subsets transferred without recalibration to five other agent families, beating random sampling at 11 of 12 budgets (1.47 against 1.86 pp at k=200). The data comes from one organization and is not released, and the paper does not measure missed regressions at an actual release gate.
Here's a paper about a repo I create to help govern agentic Spec-Driven Development and generally get far better results across the board using strict rules and structure:
https://dx.doi.org/10.2139/ssrn.7483359