N Neurarch Architectures Checks Data Docs Open the app

Resources / Datasets

Arch-Bench arena results

Frontier language models scored on design-from-spec tasks by a deterministic verifier rather than a human or an LLM judge. Each row is one model on one split: tasks passed out of tasks attempted, mean graph health score, the date of the run, and the rubric version it was measured under. 18 rows across a curated split, a procedurally generated split, and a grounded split whose designs were actually trained on a GPU. Last updated 2026-08-21.

18 model-split results CC-BY-4.0 Free to use See the board

Get it

Board dataneurarch.com/leaderboard-data.json
application/json
 
curl -s https://neurarch.com/leaderboard-data.json | jq '.sections[] | {title, rows: [.rows[] | {model, passed, total}]}'

What is in a row

model tasks passed tasks attempted mean graph health score rubric version run date

What this dataset is not

Rows carry the rubric version they were measured under and versions are not comparable. Rubric v2 included a task whose start graph already satisfied every constraint, so an empty plan passed it: a v2 row contains one free pass. Ranking a v2 row against a v3 row is the specific mistake this field exists to prevent, and we made it once ourselves.

Licence and citation

Released under Creative Commons Attribution 4.0. Cite it as:

Neurarch. Arch-Bench arena results. https://neurarch.com/d/arena.html

The rest of the set

Neurarch architecture corpus
36 neural network architectures kept as typed graphs rather than diagrams.
36 architectures · CC0-1.0
Neurarch structural check catalogue
The 41 structural checks Neurarch runs on a model graph, as data: id, severity, category, the condition that triggers it, why it costs something, and the fix.
41 checks · CC-BY-4.0
Verifier grounding study (264 graphs)
Clean reference architectures plus systematically corrupted variants (broken attention head divisibility, linear width mismatches, severed connections), each built as a real PyTorch model and run on a GPU.
264 graphs · MIT
Arch-Bench task set
The task definitions behind the benchmark: design-from-spec and repair-and-extend instances for agents that build neural network architectures.
12 curated tasks, 8 fixtures · MIT
arch-design-sft: verified architecture-design SFT data
Supervised fine-tuning data for neural architecture design treated as structured graph editing.
3,010 verified examples · MIT
Structure, verdict and trained outcome triples
The corpus that pairs what a design looks like with what it did.
80 trained graphs · MIT
Verified architecture-design reasoning traces (Claude)
Spec to reasoning to design triples where the design is re-graded by the same deterministic verifier the benchmark uses, and only passing traces are kept.
306 verified traces · MIT
Verified architecture-design reasoning traces (Grok)
The same verified spec to reasoning to design triples as the Claude split, rejection-sampled from a different frontier model, so the two can be pooled for volume or held apart to see how much of the reasoning style is model-specific.
376 verified traces · MIT