The visual app is one client. Underneath is a gym: state is a typed architecture graph, an action is a batch of edits, and the reward is a deterministic verifier · no human judge, no LLM judge, no GPU to grade. This is the loop frontier labs train reasoning models with, aimed at the one domain where the reward is machine-checkable: neural architecture itself.
RMSNorm before attention.Softmax on logits.d_model 768→1024, tie to heads.A standard reset / step / reward loop. Pure and sub-millisecond, so a rollout is cheap enough to train on.
// state = graph · action = edit batch · reward = verifier score const env = new NeurarchEnv({ maxSteps: 8, task: "classify support tickets" }) let obs = env.reset() // obs = typed graph const { reward, done, obs: next } = env.step([ { type: "add_component", componentType: "multiHeadAttention" }, { type: "add_connection", from: "norm_1", to: "attn_1" }, ]) // reward ∈ { lint, shape, cost, eval } · deterministic, un-gameable
The open-source harness (neurarch-arch-bench) serves shaped rewards over HTTP and ships a TRL GRPO loop + Colab that trains a small open model against the verifier end to end.
# reward service (zero dependencies, no build step) node env-server.mjs # shaped rewards on :8737 # train a small open model on the verifiable reward (GRPO) python training/train_grpo.py --steps 300 --count 512 --lora # baseline pass@1 on a held-out split → train → reward curve → re-eval
The 182 typed layers + 75 blocks are the actions. A verifiable env needs a defined action space; this is the hard, hand-built part a lab can't spin up overnight.
A deterministic verifier (shape, lint, serving cost, eval proxy) · not a reward model, not an LLM judge. Sub-millisecond, so rollouts are cheap.
Procedurally generated splits (same seed → identical tasks) plus edit-in-place repair where replace_model is forbidden. Reproducible with one command.
Every edit across every user is logged as a typed graph diff · the dataset of how architectures actually get built and repaired. Captured from day one so the priors sharpen the reward as it grows.
The verifier is the same deterministic function that guards every edit on the canvas, scores the arena, and shapes this reward. One environment, three surfaces.
Every number on this page comes from the same public harness. The reference provider replays a known-good solution per task, so the whole thing runs with zero dependencies and no key:
$ node leaderboard.mjs --providers=reference [PASS] reference cnn-cifar score= 80 params=19936 (1ms) [PASS] reference text-encoder score= 63 params=3971584 (0ms) [PASS] reference fix-broken-attention score= 60 params=3040200 (0ms) [PASS] reference two-tower-retrieval score= 78 params=3208192 (0ms) … -- Leaderboard -- reference 12/12 passed avg score 75
Two calibration self-tests pin the floor and the ceiling of the environment. They are the fastest way to check the reward is not gameable:
$ node calibrate.mjs --policy=noop # submit an empty plan Overall: 0/160 = 0.0% ← doing nothing passes nothing $ node calibrate.mjs --policy=reference # replay the known-good solution Overall: 160/160 = 100.0% ← every task is solvable
A task an empty plan can pass measures nothing, and inflates every score on the board equally. We shipped one — scale-under-budget asked for a bigger model while its constraints enforced only a ceiling — and a test now fails the build if a second appears.
A reward that only scores graph health cannot tell an empty graph from a real design, because an empty graph is not unhealthy. We shipped that mistake and caught it while building the reward API:
bare input → output stub score 100 grade A reward 1.00 real 7-layer CNN score 100 grade A reward 1.00 ← indistinguishable
A policy trained on that signal learns to emit nothing. The fix folds the task's own requirements into the reward, so the empty plan is penalised on every step rather than only at the final grade:
do nothing → reward -0.50 0 layers < required 4 · missing conv2d · missing linear build a CNN → reward +1.00 all requirements met
Call /api/v1/reward without requirements and the response says so in the payload, rather than letting you discover it from a policy that learned to do nothing.
Public tasks end up in training sets. A private split keeps the ability to produce an uncontaminated number permanently, and its manifest is published in full so a score is checkable without the tasks leaking. This is the manifest format, from a mint with a throwaway seed:
{
"splitId": "heldout-v1",
"generatorVersion": 2,
"count": 120,
"digest": "sha256:<64 hex>",
"families": { "gen-txf": 14, "gen-cnn": 16, "gen-gqa": 8, … },
"difficultyBiased": true
}
643 bytes against a 501 KB split. Enough to confirm two published scores came from the same task set; not enough to reconstruct one task. Difficulty is weighted by the failure modes frontier models actually hit in our recorded runs — copyable generator, uncopyable distribution. The live digest ships with the split it describes; the one above is deliberately blank, because a digest you cannot verify against is worse than none.
A submission carries action plans, never numbers. We re-apply them and grade with the verifier anyone can run from the public repo, then store our own verdict alongside the plans so every row stays re-checkable:
curl -X POST https://neurarch.com/api/v1/bench \
-d '{"model":"your-policy","results":[{"taskId":"cnn-cifar","actions":[…]}]}'
# a submitted "score" field is rejected, not ignored
# an omitted task counts as a failure → cherry-picking 3 easy tasks reads 3/12
The denominator is the benchmark, not the submission. That is the difference between a leaderboard and a press-release aggregator. Submission opens with the next deploy — the endpoint and its grading are in the repo and covered by tests; what is pending is the table it writes to. Until then, run the harness yourself with the commands above and send the JSON.
One architecture usually lives in four places at once: the design, the repo's code, the published config, and whatever is serving. Nothing keeps them in step. Answering it needs a graph on both sides — a type checker sees one file, a coding agent sees text. Real output — the published bert-base-uncased config against a small encoder written to drift from it:
major layer-count hf config has 49 layers, repo: encoder.py has 3 minor param multiHeadAttention.numHeads is 12 in hf config but 8 in repo
The 49 and the 12 are read from the real published config. The 3 and the 8 come from the encoder file, written for this illustration — the head-count drift is the case the feature exists for, not a customer finding.
Names never match across producers (self.conv1 against conv2d_0), so the comparison is name-independent by construction. It states the difference and never guesses which side is stale — only the author knows that.