A structural bug in a neural network does not fail in code review. It fails at forward time, on the GPU you already paid for, sometimes hours into a run. Neurarch keeps your model as a typed graph and runs the structural checks while you design, so the crash below never leaves the editor. This page shows the recurring bug classes and the measured evidence that the checks track reality.
We generated clean reference architectures plus systematically corrupted variants (broken attention divisibility, linear width mismatches, severed connections), built every graph as a real PyTorch model, and cross-tabulated the verifier verdict against what actually happened. 264 graphs, two seeds, torch 2.8. Reproduce it from the open benchmark repo: neurarch-ai/neurarch-arch-bench.
| Verifier verdict | n | PyTorch forward | Makes training progress |
|---|---|---|---|
| PASS (clean) | 80 | 100% ok | 90% |
| BLOCKED | 96 | 0% ok (96/96 crash) | 0% |
| Corrupted, missed by a simple rubric | 88 | 0% ok | 0% |
Read it honestly: at this scale a blocker predicts runtime failure perfectly, and clean graphs overwhelmingly construct and train. The third row is the point of this page: those are linear width mismatches that pass the open benchmark's deliberately simple rubric but still crash PyTorch. The in-product shape engine does flag them (case 2 below). "Makes training progress" means loss fell at least 20% in 60 steps; it is a trainability probe, not a quality claim.
Each case is a check that runs live in the editor. The red side is what PyTorch says when the same bug ships to the GPU.
embed_dim 768 with 10 heads. Looks fine in code review; 768 / 10 is not an integer.
shape check: embedDim % numHeads != 0 (768 % 10). Blocking. One-click fix snaps heads to the nearest divisor.
AssertionError: embed_dim must be divisible by num_heads
Crashes when the module is constructed on the training node, after the job was queued, the container pulled, and the dataset downloaded.
Upstream block outputs 512 features; the next Linear declares in_features=768. The bug class that simple linters miss (row 3 in the table above).
shape propagation: this layer's inFeatures (768) does not match the upstream output (512). One-click fix sets it from the propagated shape.
RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x512 and 768x256)
Fails on the first batch. In our measurement, 100% of graphs with this corruption failed forward.
A residual add where one branch was projected to 256 and the other stayed at 512.
merge check: both inputs to this add must have the same shape. Deterministic fix inserts the missing projection (or a reshape when only the layout differs).
RuntimeError: The size of tensor a (512) must match the size of tensor b (256) at non-singleton dimension 1
The classic skip-connection bug. It survives until the exact batch that hits the mismatched path.
numHeads=12, numKVHeads=5. GQA implementations require the query heads to group evenly over the KV heads.
GQA check: numHeads % numKVHeads != 0 (12 % 5). Blocking, with a one-click snap to the nearest valid KV head count.
assert num_heads % num_kv_heads == 0 (raises at module init or silently mis-groups, depending on the implementation)
The silent variant is the expensive one: some hand-rolled GQA code repeats KV heads incorrectly instead of failing.
A refactor deletes one edge and part of the network no longer feeds the output, or a new edge closes a cycle.
graph checks: orphan detection (a layer lands disconnected) and DFS cycle detection on every proposed edge, including edits proposed by the AI agent.
forward() fails, or worse: it runs, and the dead branch's parameters train to nothing while metrics look plausible.
In our corrupted set, severed graphs failed forward 100% of the time. The ones that do run are a slower, subtler waste.
Example arithmetic, not a promise: a 10-epoch fine-tune on one A100 80GB at $4.20/hr that dies 6 hours in wastes about $25 plus the engineer-day spent bisecting it. A multi-node pretraining run that crashes at step one wastes the whole cluster spin-up; one that trains a dead branch wastes the full budget. Inside the app, every blocking finding shows the estimated cost of the run it puts at risk, using your own GPU, epoch, and dataset settings.
Run the checks on your own architecture, free