N Neurarch Architectures Models Checks Data Docs Open the app

Resources / Datasets

Structure, verdict and trained outcome triples

The corpus that pairs what a design looks like with what it did. Each row carries a structural fingerprint of a graph (task family, parameter count, component count), the static verifier verdict on it (score and blockers), and the outcome of actually training it: the full loss curve, best and final loss, area under the loss curve, step count, and whether it converged. Every graph was trained through the same managed-GPU path the product uses, so the outcome is measured rather than simulated.

80 trained graphs MIT Free to use Source repo On Hugging Face

Get it

Triplesraw.githubusercontent.com/neurarch-ai/neurarch-arch-bench/main/triples.jsonl
application/x-ndjson
Hugging Face mirror (dataset viewer, load_dataset)huggingface.co/datasets/neurarch-ai/arch-outcome-triples
text/html
 
curl -sO https://raw.githubusercontent.com/neurarch-ai/neurarch-arch-bench/main/triples.jsonl
jq -s 'map({score: .verifier.score, bestLoss: .outcome.bestLoss})' triples.jsonl

What is in a row

task family parameter count component count verifier score blockers loss curve best loss final loss steps converged

What this dataset is not

n is 80, and the headline finding is negative: across families the correlation between verifier score and achievable quality is about zero (rho -0.02), undefined inside four families whose graphs score identically, and positive only where the score varies. It is published because that is the result. The verifier tells you whether a graph runs, not how good it will get, and this is the file that shows it.

Licence and citation

Released under MIT License. Cite it as:

Neurarch. Structure, verdict and trained outcome triples. https://neurarch.com/d/outcome-triples.html

The rest of the set

Neurarch architecture corpus
48 neural network architectures kept as typed graphs rather than diagrams.
48 architectures · CC0-1.0
Neurarch structural check catalogue
The 50 structural checks Neurarch runs on a model graph, as data: id, severity, category, the condition that triggers it, why it costs something, and the fix.
50 checks · CC-BY-4.0
Verifier grounding study (264 graphs)
Clean reference architectures plus systematically corrupted variants (broken attention head divisibility, linear width mismatches, severed connections), each built as a real PyTorch model and run on a GPU.
264 graphs · MIT
Arch-Bench arena results
Frontier language models scored on design-from-spec tasks by a deterministic verifier rather than a human or an LLM judge.
18 model-split results · CC-BY-4.0
Arch-Bench task set
The task definitions behind the benchmark: design-from-spec and repair-and-extend instances for agents that build neural network architectures.
12 curated tasks, 8 fixtures · MIT
arch-design-sft: verified architecture-design SFT data
Supervised fine-tuning data for neural architecture design treated as structured graph editing.
3,010 verified examples · MIT
Design selection replay: two trained campaigns and every selector tried
The records needed to score a design selector offline, in the shape of xRouteBench: every candidate design was trained once and the outcome kept, so a rule that picks which unexecuted design gets the GPU is evaluated by replay against the same rows as every previous rule, with no GPU and no model call.
39 trained designs · CC-BY-4.0
Who wrote the design: verifier refusal rate by author
The same pre-flight verifier applied to model designs written by three kinds of author, with the refusal rate per author kept apart.
7 rows · CC-BY-4.0
Verified architecture-design reasoning traces (Claude)
Spec to reasoning to design triples where the design is re-graded by the same deterministic verifier the benchmark uses, and only passing traces are kept.
306 verified traces · MIT
Verified architecture-design reasoning traces (Grok)
The same verified spec to reasoning to design triples as the Claude split, rejection-sampled from a different frontier model, so the two can be pooled for volume or held apart to see how much of the reasoning style is model-specific.
376 verified traces · MIT