Resources / Datasets
Who wrote the design: verifier refusal rate by author
The same pre-flight verifier applied to model designs written by three kinds of author, with the refusal rate per author kept apart. On human-written open-source model code (59 repositories, 100 parsed files, every block and warn judged by hand) the verifier found zero real defects. On agent-proposed designs, where 3 frontier models were asked for architectures from a plain sentence, it refused between 2.8% and 26.4% of what came back, depending on the model and on whether the sentence stated a real workload. One row per author, task set and recorded run: designs returned, designs refused, the share with its by-run range, how many structures our own generator could not produce, the rule ids behind the refusals, and what pre-flight quoted for the runs that did not happen.
Get it
curl -s https://neurarch.com/who-wrote-it.json | jq '.rows[] | {author, model, taskSet: .taskSet.label, returned, blocked, blockedShare, range: .blockedShareRange}'
What is in a row
author kind
model
task set
designs returned
designs refused
refusal share
by-run range
median quoted cost
What this dataset is not
Every agent row comes from sentences we wrote and models we chose, not from a customer's agent at work. A refusal is a verifier verdict that the graph would not forward-pass, not an observed crash, except where the grounding study backs the rule. The human row's zero real defects is a statement about 100 parsed files from 59 repositories read by a static parser that recognises about 41% of real model files (47 of 116 came back with ten or more layers), not about PyTorch code in general. Nothing here measures whether a design that passes trains well. The quoted dollars are A10G prices for one run of each refused design and must never be summed across rows from different task sets.
Licence and citation
Released under Creative Commons Attribution 4.0. Cite it as:
Neurarch. Who wrote the design: verifier refusal rate by author. https://neurarch.com/d/who-wrote-it.html
The rest of the set
Neurarch architecture corpus
48 neural network architectures kept as typed graphs rather than diagrams.
48 architectures · CC0-1.0
Neurarch structural check catalogue
The 50 structural checks Neurarch runs on a model graph, as data: id, severity, category, the condition that triggers it, why it costs something, and the fix.
50 checks · CC-BY-4.0
Verifier grounding study (264 graphs)
Clean reference architectures plus systematically corrupted variants (broken attention head divisibility, linear width mismatches, severed connections), each built as a real PyTorch model and run on a GPU.
264 graphs · MIT
Arch-Bench arena results
Frontier language models scored on design-from-spec tasks by a deterministic verifier rather than a human or an LLM judge.
18 model-split results · CC-BY-4.0
Arch-Bench task set
The task definitions behind the benchmark: design-from-spec and repair-and-extend instances for agents that build neural network architectures.
12 curated tasks, 8 fixtures · MIT
arch-design-sft: verified architecture-design SFT data
Supervised fine-tuning data for neural architecture design treated as structured graph editing.
3,010 verified examples · MIT
Structure, verdict and trained outcome triples
The corpus that pairs what a design looks like with what it did.
80 trained graphs · MIT
Design selection replay: two trained campaigns and every selector tried
The records needed to score a design selector offline, in the shape of xRouteBench: every candidate design was trained once and the outcome kept, so a rule that picks which unexecuted design gets the GPU is evaluated by replay against the same rows as every previous rule, with no GPU and no model call.
39 trained designs · CC-BY-4.0
Verified architecture-design reasoning traces (Claude)
Spec to reasoning to design triples where the design is re-graded by the same deterministic verifier the benchmark uses, and only passing traces are kept.
306 verified traces · MIT
Verified architecture-design reasoning traces (Grok)
The same verified spec to reasoning to design triples as the Claude split, rejection-sampled from a different frontier model, so the two can be pooled for volume or held apart to see how much of the reasoning style is model-specific.
376 verified traces · MIT