VIDRAFT Releases AX-RAY AI Safety Diagnostic Leaderboard and Evaluation Dataset on Hugging Face

TL;DR: VIDRAFT has open-sourced AX-RAY, a 117-item AI safety diagnostic framework and accompanying evaluation dataset on Hugging Face, designed to detect causal leakage — hidden, unintended causal pathways — inside general-purpose LLMs. Unlike standard performance benchmarks, AX-RAY maps safety failure modes to real-world legal and regulatory frameworks across multiple jurisdictions. If you work on model evaluation, red-teaming, or AI safety, this is worth a look.

What it is

AX-RAY is a safety diagnostic leaderboard and evaluation dataset published by VIDRAFT (비드래프트), a Korean deep-tech AI startup, on August 13, 2026. It is explicitly positioned as something different from a capability benchmark: rather than measuring what a model can do, it probes where a model might behave in unsafe, unintended, or legally non-compliant ways.

Key properties straight from the source:

• 117 diagnostic items covering potential risk behaviors and vulnerabilities inside general-purpose AI models
• Causal leakage detection — the framework targets a specific failure phenomenon where a model's decisions are influenced by hidden information or unintended causal cues rather than the intended reasoning path
• Jurisdiction-mapped items — each of the 117 items is linked to existing laws and regulatory frameworks in specific countries, making compliance-aware evaluation tractable
• Cultural and religious norm coverage — for the Arabic-speaking world, the framework goes beyond local statutes and maps items to Sharia-derived norms, enabling culturally contextual safety assessment
• Public leaderboard — results for evaluated models are published openly, including a finding of causal leakage signals in two models, one of which is an NVIDIA general-purpose AI model

This combination of structured diagnostic items, legal mapping, and public leaderboard results makes AX-RAY a framework aimed at practitioners who need defensible, jurisdiction-aware safety evidence — not just aggregate safety scores.

How it works

At a conceptual level, AX-RAY approaches AI safety diagnosis in two layers:

1. Behavioral probing for causal leakage
Causal leakage is the phenomenon where a model's output is driven not by the legitimate reasoning chain you'd expect, but by latent or spurious causal signals embedded in training data or prompts. Think of it as a form of shortcut learning that manifests as unpredictable, potentially dangerous behavior — the kind that might cause an autonomous agent to bypass a safety guardrail or take an unexpected action. AX-RAY structures its 117 evaluation items to surface these hidden dependencies in general-purpose LLMs in a reproducible way.

2. Regulatory and normative mapping
Each diagnostic item isn't free-floating — it is anchored to a concrete legal or social norm from a target jurisdiction. This means evaluation outputs are interpretable in terms of actual compliance risk, not just abstract safety scores. The Arabic-world extension, which incorporates religious and social norms alongside civil law, is a concrete example of this design principle applied to a non-Western regulatory context.

The result is a diagnostic pipeline where a developer or safety auditor can run a model against the dataset, get a per-item breakdown, and then directly trace any flagged items to the specific legal or normative standard that may be implicated.

Benchmarks & results

Public numbers and findings reported in the source:

• 2 models showed detectable causal leakage signals in AX-RAY evaluation, including one NVIDIA general-purpose AI model
• VIDRAFT's own foundation model series (AETHER / Darwin) has accumulated 1 million+ downloads on Hugging Face
• The AETHER line has achieved #1 on the K-AI leaderboard, 90.9% on GPQA, and #1 on the Fast Gemma Challenge

Note: the benchmark results listed above for VIDRAFT's own models relate to their foundation model family, not to AX-RAY itself. The AX-RAY-specific finding is the causal leakage detection across the two evaluated models.

How to try it

AX-RAY and its evaluation dataset are publicly available on Hugging Face as of August 13, 2026. You can browse the leaderboard and download the dataset directly from VIDRAFT's Hugging Face organization page. No specific dataset slug or CLI command has been officially published in the source at time of writing, but the standard Hugging Face discovery path applies:

1. Go to huggingface.co and search for VIDRAFT or AX-RAY
2. Locate the dataset card and leaderboard entries under VIDRAFT's organization
3. Use the standard huggingface-cli or datasets library to pull the evaluation data once you've identified the correct repository slug from the official page

No API endpoint, GitHub repo URL, or specific model card name was disclosed in the source — use the Hugging Face search interface to find the canonical links rather than guessing paths.

FAQ

Q: Is AX-RAY a general-purpose safety benchmark I can drop into my existing eval pipeline?
A: It is designed as a diagnostic tool rather than a single-score benchmark. The 117-item structure and legal-mapping layer mean you get granular, jurisdiction-tagged findings rather than one aggregate safety number — more useful for compliance and red-teaming workflows, but it requires interpreting item-level results.

Q: What exactly is "causal leakage" and why does it matter for production LLMs?
A: Causal leakage refers to a model acting on hidden or unintended causal signals rather than the expected reasoning path. In production, this can manifest as guardrail bypasses or unexpected autonomous actions. AX-RAY is one of the first public frameworks attempting to detect and reproduce this phenomenon at scale in general-purpose LLMs, which is why the finding in the NVIDIA model is notable.

Q: Does AX-RAY cover non-English or non-Western regulatory contexts?
A: Yes. The framework explicitly maps diagnostic items to multi-jurisdictional legal frameworks, and includes Arabic-world coverage that extends to religious and social norms — making it one of the few public safety benchmarks with built-in cross-cultural regulatory scope.

Originally reported by ZDNet Korea (2026-08-14) — source article.