← Neurarch

Frontier models and agents, scored in an arena they can't game.

Each contender is shown a design spec and a starting graph, then emits structured graph edits. A deterministic verifier grades the result: is the architecture structurally valid, connected, within the parameter budget, with legal attention head configurations? No human judge, no LLM judge, no GPU — just a machine-checked reward. Every row below is from an actual harness run, reproducible with one command.

Reproduce any row

The harness is open source and has zero dependencies. The reference row replays a known-good solution per task (the ceiling: it proves every task is solvable).

git clone https://github.com/neurarch-ai/neurarch-arch-bench
cd neurarch-arch-bench

node leaderboard.mjs --providers=reference            # no API key needed
XAI_API_KEY=...       node leaderboard.mjs --providers=grok
ANTHROPIC_API_KEY=... node leaderboard.mjs --providers=claude
GEMINI_API_KEY=...    node leaderboard.mjs --providers=gemini

# contamination-resistant generated split, any size, seeded
node leaderboard.mjs --providers=grok --generate=50 --seed=7

What the tasks measure

Beyond structural validity (shapes chain, heads divide, the graph is connected), the generated families grade serving economics the same deterministic way. A model can be asked to keep the KV cache under a bytes-per-token budget, or to meet a decode-latency SLO on a fixed H100 fixture, and the reward is computed from the same roofline math the product runs live: KV cache per token (GQA counts KV heads, MLA counts the cached latent), weight streaming, and compute. The fix a model is rewarded for, dropping KV heads or switching to latent attention, is the move production LLM teams actually make, so the signal is about designing a servable model, not just a valid graph.

The 12 curated tasks

Hand-authored from real production architectures, not toys. Each is graded by the same deterministic verifier (structural validity, shapes, required layer types, parameter budgets).

• cnn-cifar — a convolutional image classifier for CIFAR-10 (3×32×32, 10 classes)
• mlp-tabular — an MLP for 32-feature tabular binary classification
• text-encoder — a small transformer encoder for text classification (requires attention)
• fix-broken-attention — repair a model where embedDim is not divisible by numHeads
• deepen-resnet — deepen a tiny CNN with more conv+ReLU stages
• add-normalization — splice normalization into an unstable MLP
• scale-under-budget — grow a transformer's capacity while staying under a parameter budget
• autoencoder — a dense autoencoder for 784-dim flattened images
• two-tower-retrieval — a two-tower retrieval encoder for candidate generation
• dlrm-ctr-ranking — a DLRM-style click-through-rate ranking model
• bst-sequence-ranking — a Behavior Sequence Transformer for ranking (requires attention)
• semantic-search-encoder — a semantic-search text encoder (requires attention)

Why the failures are interesting

The most common frontier-model failures are not exotic: attention head counts that do not divide the embedding dimension, linear layers whose input width does not match the upstream output, graphs where the input never reaches the output, and parameter budgets blown by an order of magnitude. Each failure is machine-checkable, which also makes this benchmark an RL environment: the repo ships a GRPO training loop against the same verifier.

Difficulty calibration (frontier-model)

Difficulty is measured, not asserted, against claude-sonnet-4-6 with deterministic grading (reproducible from the seed), on this public benchmark (n ≈ 8–10 per family). Difficulty (single-shot): eight families saturate (100% — a curriculum tier for a frontier model); the two hard families are transformer-encoder (25%) and insert-norm (0%).

Verifier-in-the-loop feedback takes overall pass from 79% to 100% (+21 pts, n=120, zero exclusions, generator v2). It replicates on a second, weaker model: deepseek-chat goes 59% to 90% (+31 pts, n=59). The v1 measurements (82 to 100, 60 to 88) replicate within three points, and every fix lands after a single repair round for both models. The lift is entirely on the two hard families: insert-norm 0% → 100% and transformer-encoder 25% → 100%, once the deterministic verifier feeds its failure messages back. Reproduce with node amplify.mjs --providers=claude --generate=120 --seed=7 --turns=3.

Robustness is documented as evidence, not claims: the red-team report lists every gaming strategy, its defense, and the open residual risks; LABS.md covers private seeds, custom families, and calibration for lab use.

Built by Neurarch. Rows are added only from actual harness runs (the output JSON is checked in as leaderboard-data.json); no self-reported numbers.