Data / Calibration

A passing score does not rank two legal designs

Our verifier proves a design can train. It does not tell you which of two designs that both pass is the better one to spend a GPU on, and we can put a number on how badly. This page is that number and the rows it comes from.

Designs

All passed, all trained

24/24

Outcome spread

6.8% to 97.2%

Spearman

0.17

Distinct static scores

One campaign, 2026-08-21, every design trained end to end on a managed T4 under identical budgets and configs. Every number on this page is computed from the campaign rows below, not written by hand.

The spread

20 of the 24 designs are scored on validation accuracy. They finished between 6.8% and 97.2%. Every one of them passed the verifier first.

	Task	Dataset	Designer	Static score	Trained	Params
1	cifar-cnn	cifar10	hand-authored reference	100	62.1%	255,978
2	cifar-cnn	cifar10	xai	100	64.0%	261,034
3	cifar-cnn	cifar10	openrouter	100	6.8%	1,070,794
4	cifar-cnn	cifar10	groq	100	66.1%	521,162
5	cifar-deepen	cifar10	hand-authored reference	100	63.0%	255,978
6	cifar-deepen	cifar10	xai	100	66.4%	456,234
7	cifar-deepen	cifar10	openrouter	100	65.6%	834,730
8	cifar-deepen	cifar10	groq	100	64.3%	456,234
9	tabular-mlp	tabular	hand-authored reference	100	loss 0.0041	5,569
10	tabular-mlp	tabular	xai	100	loss 0.0024	3,457
11	tabular-mlp	tabular	openrouter	100	loss 0.0026	13,057
12	tabular-mlp	tabular	groq	100	loss 0.0030	3,457
13	mnist-mlp	mnist	hand-authored reference	100	96.8%	235,146
14	mnist-mlp	mnist	xai	100	96.8%	235,146
15	mnist-mlp	mnist	openrouter	100	97.2%	567,434
16	mnist-mlp	mnist	groq	100	96.9%	235,146
17	cifar-deep	cifar10	hand-authored reference	100	58.8%	82,330
18	cifar-deep	cifar10	xai	100	17.8%	307,498
19	cifar-deep	cifar10	openrouter	97	16.9%	402,026
20	cifar-deep	cifar10	groq	100	9.9%	363,882
21	mnist-tiny	mnist	hand-authored reference	100	91.8%	7,165
22	mnist-tiny	mnist	xai	100	91.9%	7,165
23	mnist-tiny	mnist	openrouter	100	90.8%	6,370
24	mnist-tiny	mnist	groq	100	91.1%	7,980

Why the correlation is near zero

Not because the outcomes are noise. Because the score has almost no range: a checker that returns the same verdict for everything it passes cannot order anything.

Static score	Designs
100	23
97	1

Two poolings, both reported

Rank correlation between static score and trained outcome, computed two ways on the same rows.

0.17Raw outcomes pooled across tasks and ranked once. This is the number we quote, and it is the one stored in the campaign artifact.
It flatters nobody, but it is not scale-free: every MNIST row outranks every CIFAR row before the static score is consulted at all.

0.09Ranks taken inside each task, then pooled. Task scale cannot leak in.
Lower than the number we quote. We publish it because picking the flattering pooling and not mentioning the other one is the exact failure this page exists to not commit.

What this page does not claim

not yetThat we have closed the gap. A rule mined from these outcomes moves the correlation to 0.41 when the same graphs are rescored, but that is in-sample: the rule was derived from the outcomes it is being scored against. The out-of-sample number, on a campaign the rule has never seen, does not exist yet.

read it rightThe four tabular rows are scored on validation loss, so they carry no accuracy. Reading their accuracy field as 0% would widen the spread to 0% to 97.2% in the direction that flatters us, which is why the spread above is taken over the 20 accuracy-scored rows only.

one campaign24 designs, 6 tasks, one day, one GPU class. It is enough to show a static score does not rank legal designs. It is not a benchmark of anybody's model.

Every selector that has been tried, on the same rows

The question a ranking is for: given designs nobody has run, which one gets the GPU? Below, every rule that has picked a design here is scored by replay against the trained outcomes above and against the out-of-sample campaign: the static score, the static score with a size tie-break, always the smallest design, always the largest design, and two frontier models reading the exported code. A selector sees the task, the designer, the static score and the parameter count, and may abstain. Nothing here is large; read the sample sizes first.

In-sample campaign (2026-08-21, T4)

in-sample: 24 trained designs over 6 tasks. Pairwise preference within a task; a tie is scored as 0.5. A row with fewer than 10 decided pairs is marked not quotable.

Selector	Kind	Decided / judged	Correct / wrong	Accuracy, ties as 0.5	When decided	Headroom captured	Quotable
static-verifier	verifier	3 / 36	2 / 1	51.4%	66.7%	2.1%	no
static-then-smallest	verifier	30 / 36	14 / 16	47.2%	46.7%	-7.7%	yes
smallest-params	trivial	30 / 36	14 / 16	47.2%	46.7%	-7.7%	yes
largest-params	trivial	30 / 36	16 / 14	52.8%	53.3%	-2.5%	yes

Out-of-sample campaign (2026-09-01, T4)

out-of-sample: 15 trained designs over 5 tasks. Pairwise preference within a task; a tie is scored as 0.5. A row with fewer than 10 decided pairs is marked not quotable.

Selector	Kind	Decided / judged	Correct / wrong	Accuracy, ties as 0.5	When decided	Headroom captured	Quotable
static-verifier	verifier	4 / 15	3 / 1	56.7%	75.0%	21.8%	no
static-then-smallest	verifier	15 / 15	5 / 10	33.3%	33.3%	0.6%	yes
smallest-params	trivial	15 / 15	4 / 11	26.7%	26.7%	-40.5%	yes
largest-params	trivial	15 / 15	11 / 4	73.3%	73.3%	66.9%	yes
judge:claude-opus-5	judge	15 / 15	12 / 3	80.0%	80.0%	84.7%	yes
judge:grok-4.3	judge	15 / 15	11 / 4	73.3%	73.3%	68.3%	yes

read it rightOut of sample, always picking the larger design decides all 15 pairs and gets 11, the same as one of the two frontier models and one behind the other. A judge's number is not evidence that reading code sees what size does not until it beats that row on a split minted after it was fit.

replay itBoth campaigns, the judges' verdicts and this board are published as files (the dataset page) and on Hugging Face, so a selector of yours is scored on identical rows with no GPU and no model call.

Check it yourself

The rows above are the whole campaign, published as data. Both correlations, the histogram and the spread are derived from them, so a reader who disagrees with our statistic can compute their own.

The 24 rows as JSON All datasets The checks being scored POST /api/v1/check