LLM Watch Weekly: The Measurement Problem
Get ahead of the curve with LLM Watch
Welcome, Watcher! This week in LLM Watch:
Enabling web search on ChatGPT reduced benchmark accuracy by up to 8 percentage points, and repeated runs of the same prompt produced inconsistent answers on up to 21% of prompts - raising hard questions about how we evaluate deployed AI systems.
A neuro-symbolic RAG framework that compiles retrieved text into executable Prolog modules achieves 61.1% accuracy on ShARC, outperforming a standard RAG baseline’s 42.8% - without any domain-specific training.
A new benchmark of 201,247 real investor decisions reveals that leading LLMs produce logically plausible financial reasoning (scoring ~4/5) but ground that reasoning in actual market events at a dismal 0.8 - 2.8/5 - polished language masking shallow understanding.
Let’s dive in.
What Current AI Benchmarks Leave Unmeasured: Modality, Search, Citations, and Implications (for Safety Evaluations)
What problem does it solve?
When we evaluate LLMs for safety and reliability, we almost always do it the same way: send prompts through an API, run each prompt once, and report accuracy. But that’s not how most people actually use these models. Real users interact through chat interfaces, often with web search enabled, and they ask the same kinds of questions repeatedly. The gap between how we test and how people use these systems is a blind spot - and it’s one that matters most precisely in the safety-critical contexts where benchmarks are supposed to give us confidence.
The researchers ask a simple question: if you test the same model through different access modalities (ChatGPT’s chat UI vs. OpenAI’s API), with and without web search, and across multiple runs, do you get the same results? If not, what are we actually measuring when we report a single accuracy number?
How does it solve the problem?
The team took a stratified sample of 401 prompts from two widely-used benchmarks - BBQ (measuring social bias) and SafetyBench (measuring safety) - and collected 4,812 total responses across both modalities, with and without web search, running each prompt three times. This is a straightforward but labor-intensive audit design: hold the model family constant, vary the conditions that differ between lab evaluation and real deployment, and measure what changes.
Beyond standard accuracy, they measure response consistency across repeated runs, text similarity between modality pairs, citation grounding (what sources does the model actually cite, and do they differ by modality?), and abstention behavior (when does the model refuse to answer?). These are the kinds of behavioral variations that a single accuracy number completely obscures.
The design is intentionally conservative - same model family, same prompts, same benchmarks the community already trusts. The only things that change are the access conditions. That makes any observed differences hard to dismiss as artifacts.
What are the key findings?
Chat UI responses were less accurate than API responses on both benchmarks with search disabled. Enabling web search reduced accuracy by up to 8 percentage points and, in one case, actually reversed which modality performed better - the chat UI went from underperforming to outperforming the API on SafetyBench when search was turned on.
Consistency was another concern. Repeated runs of the same prompt produced different responses in up to 21% of cases. The two modalities grounded their answers in different citations even when answering the same question. Abstention behavior - whether the model refuses to answer - was also inconsistent across modalities and search conditions.
Why does it matter?
If a model’s safety profile changes depending on whether you test it through the API or the chat interface - and changes again when web search is toggled on - then single-condition evaluations are giving us a false sense of precision. The model your users interact with through a chat interface with search enabled may behave meaningfully differently from the model you evaluated through the API without search. That delta is where real-world risk lives, and current benchmarking practices are largely blind to it.
Evaluating Investment Logic in Large Language Models: A Real-World Benchmark Towards Personalized Financial Agents
What problem does it solve?
Financial AI benchmarks tend to evaluate LLMs in one of two ways: static question answering (can the model answer finance trivia?) or terminal profit-and-loss (did the model make money?). Neither captures what actually matters for a financial agent. Static QA ignores the fact that the same market signal should trigger different actions for different investors - a retiree and a 25-year-old tech worker should not respond identically to the same earnings report. And P&L-based evaluation can’t distinguish between a well-reasoned trade and a lucky one.
The deeper problem is that investment competence is inherently personalized. A model that produces fluent, logically structured reasoning about a stock might still be completely wrong for a specific investor’s risk profile, time horizon, or portfolio constraints. Current benchmarks have no way to measure this.
How does it solve the problem?
The authors introduce InvestLogicBench, a benchmark built from 201,247 documented decisions made by 151 real-world investors. Each decision is structured as a five-stage trace they call P→E→R→D→O: investor Profile, observable market Events, investment Reasoning, executable Decision, and delayed Outcome. This isn’t synthetic data - these are real investment decisions with real outcomes, decomposed into inspectable reasoning chains.
The benchmark supports three evaluation modes: comprehension (can the model understand a given reasoning trace?), profile-conditioned generation (can it produce reasoning appropriate for a specific investor profile?), and end-to-end replay (can it reconstruct the full decision process?). The key design insight is separating process quality from outcome quality. A model might arrive at a profitable trade through sloppy reasoning, or make a sound decision that happens to lose money due to market conditions. InvestLogicBench can distinguish these cases.
The P→E→R→D→O structure also functions as a proposed data-system interface for consequential agents more broadly. The authors argue that any agent making high-stakes personalized decisions needs versioned profiles, temporal provenance for information, inspectable retrieval, decision ledgers, and replayable outcomes.
What are the key findings?
Across four leading LLMs, the results reveal a telling pattern: logical plausibility scores remain near 4 out of 5, but event grounding - whether the model’s reasoning is actually anchored in the specific market events it references - drops to just 0.8 - 2.8 out of 5. In other words, the models produce reasoning that sounds right but isn’t reliably connected to the evidence it claims to use.
Return quality and process quality also disagree, confirming the authors’ thesis that outcome-only evaluation hides important failure modes. A model can generate polished, logically coherent investment reasoning that would score well on any fluency metric while being poorly grounded in the actual market conditions it’s supposed to be analyzing.
Why does it matter?
This paper exposes a failure mode that extends well beyond finance: LLMs are increasingly good at producing reasoning that looks right without being right. In domains where decisions are personalized and consequential - healthcare, legal advice, financial planning - fluent reasoning can be actively dangerous if it isn’t grounded in the specific evidence and user context that should drive the decision.
The P→E→R→D→O framework is worth watching as a general pattern for evaluating agentic systems. Any domain where the same evidence should produce different actions for different users needs this kind of process-level evaluation. Outcome-based metrics alone will continue to hide the gap between articulate reasoning and grounded reasoning.
NeSy-RAG: Neuro-Symbolic RAG for Explainable Question Answering
What problem does it solve?
Standard RAG pipelines retrieve relevant text chunks and feed them to an LLM to generate an answer. This works well enough for straightforward factual questions, but the reasoning process remains opaque. You can’t easily verify which retrieved evidence actually supported which part of the answer, and you can’t trace the logical steps the model took to get there. When the answer depends on user-specific conditions - “Am I eligible for this benefit given my circumstances?” - standard RAG has no systematic way to detect that it’s missing critical information. It either guesses or hedges.
This is a real problem for any application where the answer depends on conditional logic: regulatory compliance, eligibility determination, policy interpretation. The user needs to know not just the answer, but why, and what assumptions the system made to get there.
How does it solve the problem?
NeSy-RAG introduces a modular neuro-symbolic pipeline that sits between retrieval and answer generation. Instead of passing retrieved chunks directly to an LLM for free-form generation, the system compiles each chunk into a Prolog module containing semantically meaningful predicates - essentially Boolean claims that may depend on user-specific facts. Picture it as translating natural language rules into executable code: “If the applicant is over 65 AND has lived in the UK for 3+ years, then they qualify.”
The system uses joint natural language-code embeddings to retrieve relevant predicates and compose them into Prolog queries. The framework includes a symbolic knowledge-gap detection mechanism that identifies missing user facts whose truth values would change the query outcome. If the system can’t determine eligibility without knowing the user’s age, it automatically triggers a follow-up question rather than guessing.
Executing the resulting Prolog queries produces deterministic answers with transparent execution traces. Every reasoning step links back to its originating source chunk, giving you full attribution - not the probabilistic “this chunk was probably relevant” attribution of standard RAG, but logical, verifiable provenance.
What are the key findings?
On the ShARC benchmark - a conversational QA dataset that requires conditional reasoning and follow-up questions - NeSy-RAG achieves 61.1% accuracy without any domain-specific training. A same-model RAG baseline achieves 42.8%, making this an 18.3 percentage point improvement from the symbolic reasoning layer alone. This improvement comes entirely from the structural change in how reasoning is performed - no additional training data, no fine-tuning, no larger model. The Prolog compilation step converts a fuzzy generation problem into a crisp logical execution problem, and the accuracy gains follow directly from that conversion.
Why does it matter?
For anyone building RAG systems that need to handle conditional logic - eligibility checks, compliance verification, policy interpretation - NeSy-RAG offers a concrete architectural pattern. The key insight is that separating retrieval, logic compilation, and execution gives you verifiability and gap detection that end-to-end generation can’t provide.
In production RAG systems, one of the hardest failure modes to catch is when the system confidently answers a question it doesn’t have enough information to answer correctly. NeSy-RAG’s symbolic layer makes missing information explicit and actionable, turning a silent failure into a follow-up question. That’s a meaningful improvement for any high-stakes QA application.
TS-RAG: Retrieval Augmented Generation for Time Series Forecasting
What problem does it solve?
Retrieval-Augmented Generation (RAG) has proven its value in Natural Language Processing (NLP) by letting language models pull in relevant external information at inference time. The natural question is whether the same idea works for time series forecasting: if you’re predicting future values of a sequence, could retrieving similar historical patterns improve accuracy? In principle, yes - but in practice, the approach doesn’t transfer cleanly.
The core difficulty is that time series models aren’t language models. They typically have smaller parameter counts, limited training data, and lack the generative flexibility that makes RAG work so well in NLP. Simply concatenating a retrieved reference sequence into the input - the equivalent of stuffing context into a prompt - doesn’t reliably help. The model doesn’t know how to weight the reference against its own input, and the concatenation can introduce noise rather than signal.
How does it solve the problem?
TS-RAG introduces a dedicated fusion mechanism for incorporating retrieved sequences into time series forecasting. Rather than treating reference sequences as additional input tokens (the naive approach), the framework introduces specially designed reference tokens that mediate between the input sequence and retrieved similar sequences.
The retrieval step identifies historically similar time series patterns from a reference database. The fusion step is where TS-RAG departs from standard RAG: instead of concatenation, the reference tokens act as an information bridge, allowing the model to selectively incorporate temporal dynamics from the retrieved sequences. Think of it as giving the model a structured way to say “this historical pattern is relevant to what I’m seeing now” rather than forcing it to figure out what to do with a raw appended sequence.
This design acknowledges a fundamental asymmetry between language and time series domains. In language, context is context - more relevant text generally helps. In time series, the relationship between a reference pattern and the current input is more nuanced: the reference might be similar in shape but different in scale, or similar in periodicity but offset in phase. The reference token mechanism gives the model a learnable interface for handling these relationships.
What are the key findings?
The authors report that TS-RAG achieves consistent state-of-the-art performance across several real-world forecasting benchmarks. While the abstract doesn’t break out individual benchmark numbers, the emphasis on consistency is notable - many time series methods show strong results on one or two datasets but degrade on others. Consistent improvement across benchmarks suggests the reference token mechanism generalizes rather than overfitting to specific data characteristics.
The key result is less about any single metric and more about the proof of concept: RAG can work for time series forecasting, but only if you design the fusion mechanism for the domain rather than importing it wholesale from NLP. The naive concatenation approach that the authors contrast against serves as a useful negative result - it confirms that the problem isn’t retrieval quality but information integration.
Why does it matter?
If you’re working on forecasting problems where historical analogues exist - energy demand, financial markets, sensor networks - TS-RAG suggests that retrieval-augmented approaches can improve accuracy, provided you invest in the right fusion architecture.
More broadly, this is part of a growing trend of adapting LLM-native techniques (RAG, in-context learning, chain-of-thought) to non-language domains. The lesson from TS-RAG is that these adaptations require domain-specific engineering at the interface layer. The retrieval concept transfers - the implementation details don’t.
Benchmarking and Enhancing LLMs for Rule-Intensive Review of National Standard Documents
What problem does it solve?
National standard documents - like China’s GB/T standards - are lengthy, highly structured documents governed by explicit rules about scope, terminology, normative wording, and cross-section consistency. Reviewing these documents for compliance is currently a manual, expert-driven process that’s expensive and difficult to scale. It’s the kind of task that looks like it should be easy for LLMs: the rules are explicit, the documents are structured, and the errors are well-defined. But existing benchmarks don’t actually test this capability.
Most LLM benchmarks for document understanding focus on domain knowledge or question answering. They don’t test whether a model can systematically audit a document against a complex rule set, identify specific violations, locate them precisely, and classify them correctly. That’s a different and harder task - more like a code linter for natural language documents than a reading comprehension test.
How does it solve the problem?
The authors introduce GB/T-Bench, the first benchmark specifically designed for structured document review. At its core is the GB/T Review Taxonomy, a hierarchical schema covering document structure, scope alignment, normative modality, terminology consistency, and normative references, with 25 diagnosable error types. This taxonomy defines what “correct” looks like for a national standard document.
To generate evaluation data, the team developed a controllable counterexample generation mechanism. Starting from 488 real documents, they combine deterministic rule-based mutations with constrained LLM rewriting to produce 7,306 traceable review error instances. Each error has a known location, dimension, and type - making evaluation precise rather than subjective.
The evaluation protocol itself requires exact matches on error location, review dimension, and error type, plus document-level coverage metrics. This is significantly harder than “did the model notice something was wrong?” - it’s “did the model identify the right error, in the right place, for the right reason?”
The authors also propose GB/T-Reviewer, a multi-agent framework that converts review knowledge into specialized skills and coordinates global inspection, targeted diagnosis, rule scanning, and result verification. Each agent handles a different aspect of the review process, and their outputs are composed into a comprehensive document audit.
What are the key findings?
The results reveal a substantial gap between human experts and current LLMs. The strongest single model achieves a CMCS (Comprehensive Multi-dimensional Compliance Score) of only 0.3280, compared to 0.6640 for human experts - roughly half the expert-level performance. The GB/T-Reviewer multi-agent framework raises the best score to 0.5094, a meaningful improvement but still well below expert level.
Fourteen mainstream LLMs were evaluated, and the consistent finding across all of them is that rule-intensive document review remains a hard problem. Models can catch some errors but struggle with the systematic, exhaustive coverage that expert reviewers achieve. The multi-agent approach helps by decomposing the task into specialized sub-tasks, but the gap suggests fundamental limitations in how current models handle complex, interacting rule sets applied to long documents.
Why does it matter?
Many organizations are exploring LLMs for compliance, auditing, and document review tasks, but without rigorous evaluation instruments, it’s hard to know whether the models are actually ready. GB/T-Bench provides a clear, quantitative answer: not yet, but structured multi-agent approaches can close roughly half the gap.
The broader lesson is about task decomposition. Rule-intensive review requires a model to hold a complex rule set in mind while scanning a long document - a combination of long-context understanding and systematic reasoning that remains challenging. The multi-agent approach works precisely because it breaks this compound task into manageable pieces. For practitioners considering LLMs for compliance or quality review, the takeaway is that single-model approaches will likely underperform, but well-designed multi-agent pipelines can get meaningfully closer to expert-level performance.
Papers of the Week
Brief highlights from other notable papers this week:
Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents - Introduces a reference framework for assessing the quality of benchmarks used to evaluate task-oriented conversational agents. The work highlights that poor benchmarks with inconsistent tasks and limited policy coverage can lead to unreliable agent evaluations - a meta-evaluation problem that mirrors this week’s broader theme.
MetaboLLM: A Metabolomics-Specialized Large Language Model for Biochemical Knowledge Integration and Predictive Metabolite Graph Construction - Adapts an LLM for metabolomics through continual pretraining, supervised fine-tuning, and structured retrieval, enabling predictive metabolite graph construction from heterogeneous biochemical data sources. A compelling example of domain-specialized LLM adaptation beyond typical NLP tasks.
Beyond Sequence Order: Syntax-Informed Positional Embeddings for Transformers - Proposes SiPE (Syntax-informed Positional Embeddings), which learns a lightweight syntactic prior from dependency parse trees to supplement standard positional encodings. The approach encodes structural relationships between tokens rather than just sequential distance.
SAGA: Score-Weighted Adaptive Generation Alignment for Low-Resource Nordic Language Models - Presents a preference optimization method that works without costly human preference annotations, targeting morphologically rich, low-resource Nordic languages. SAGA uses score-weighted alignment to bridge the annotation scarcity gap for languages where preference data is minimal.
Learning When to Trust via Selective Context Preference Optimization - Tackles the nuanced problem of teaching models when to trust external context and when to ignore it. The key insight: a model trained to blindly resist misleading context becomes useless when the context is actually correct, so selective trust calibration is essential.
RP-OPSD: Reasoning-Pivot-Guided On-Policy Self-Distillation for Multilingual Reasoning Transfer - Extends reasoning capabilities to non-English languages using on-policy self-distillation with reasoning pivots, providing dense token-level supervision on student-generated rollouts for more effective multilingual reasoning transfer.
Visual Grounding in Zero-Shot Vision-Language Control - Investigates whether vision-language models used as zero-shot controllers actually ground their decisions in visual input. The finding is sobering: simulator dynamics and conservative action priors can produce favorable scores without meaningful perception, calling into question how we evaluate visual grounding.
Dynamic Graph Prompting via Topology-Routed Mixed-Curvature Experts - Reveals that temporal shifts in graph topology require different embedding geometries and proposes routing between mixed-curvature expert spaces for dynamic graph prompting. Adapts pre-trained temporal backbones to label-scarce downstream tasks using lightweight prompts.
Putting It All Together
This week’s papers converge on a single uncomfortable theme: we are often measuring the wrong things, or measuring the right things under the wrong conditions. The benchmark audit from Encarnación et al. shows that the same model tested through different access modalities produces meaningfully different safety results. InvestLogicBench reveals that LLMs can score well on logical plausibility while being poorly grounded in the evidence they claim to use. GB/T-Bench demonstrates that even explicit, rule-based review - a task that should play to LLM strengths - exposes a 50%+ performance gap between models and human experts. And in the quick hits, the visual grounding paper finds that VLM controllers can succeed without actually perceiving their environment. Our evaluation frameworks are systematically optimistic because they test under idealized conditions and measure surface-level outputs.
The second theme is the growing sophistication of RAG architectures. We’re moving well past “retrieve and concatenate.” NeSy-RAG compiles retrieved text into executable Prolog programs, gaining verifiability and gap detection. TS-RAG introduces domain-specific fusion tokens because naive concatenation fails for time series. Both papers suggest that the next generation of RAG systems will be defined not by retrieval quality but by how intelligently they integrate retrieved information - and that this integration layer needs to be designed for the specific domain and reasoning type.
Looking ahead, I expect the measurement problem to intensify as models are increasingly deployed through agentic pipelines with search, tool use, and multi-turn interaction. The gap between API-based benchmarking and real-world behavior will only widen, and the community will need evaluation frameworks that account for the full deployment stack - not just the model in isolation. The work we discussed today is laying some of the conceptual groundwork for that shift. Until next week!