Your LLM Can Score Well and Still Violate Causal Inference: What AX-Ray Found
Most LLM benchmarks test the final output. But what if the model reaches that output through a computation path that violates autoregressive causality?
VIDRAFT's AX-Ray, powered by FINAL-Bench Diagnostics, reports a public case involving two general-purpose models:
nvidia/Nemotron-H-8B-Base-8K
Zyphra/Zamba2-1.2B
AX-Ray identified, reproduced, and demonstrated a causal-leakage defect in both.
The invariant an autoregressive model should preserve
At token position t, the model should only depend on tokens available at positions <= t.
Conceptually:
prefix_behavior(x[0:t])
should remain unchanged
when only x[t+1:] is modified
If changing an inaccessible suffix changes prefix hidden states, logits, or related scoring behavior, future information has leaked into the prefix computation.
That is different from:
- hallucination
- prompt injection
- refusal or alignment failure
- benchmark contamination
Those issues concern outputs, behavior, or data. Causal leakage concerns the correctness of the inference path itself.
Why a capability leaderboard may not catch it
A QA, coding, or math benchmark usually evaluates the final answer. A model can therefore remain competitive while a deeper invariant is broken.
A deployment diagnostic needs to test questions such as:
- Does prefix behavior remain invariant when only the suffix changes?
- Are cached and non-cached serving paths causally consistent?
- Does tokenizer or serving behavior alter the expected computation?
- Should a critical structural defect override the aggregate score?
AX-Ray answers the last question with an explicit policy: confirmed causal leakage triggers an F-gate, regardless of aggregate capability performance.
How AX-Ray structures the assessment
AX-Ray currently exposes:
- 3 axes: MODEL-SCAN, AX-SCAN, AGENT-SCAN
- 11 operational categories
- 117 public diagnostic item records
The categories cover causal and serving integrity, reliability, robustness, security and alignment, data integrity, architecture and efficiency, internal structure, remediation, serving infrastructure, infrastructure security, and compliance.
The point is not to create one more opaque score. Each diagnostic item connects a technical question to evidence, severity, detection direction, remediation direction, and governance context.
A note on responsible disclosure
The public report names the affected models and explains the defect class and its safety significance. It does not disclose proprietary probe recipes, internal thresholds, raw adversarial prompts, sensitive outputs, or implementation details that could facilitate misuse or compromise the diagnostic method.
The practical takeaway
Capability tests answer:
Can this model solve the task?
Deployment diagnostics must also answer:
Is the model—and the serving path around it—computationally trustworthy?
We need both. A strong final-answer score should not be allowed to average away a structural failure.
Read the complete case and methodology overview:
Top comments (0)