Checks / structure
LM head width disagrees with embedding vocab
Check R34. Runs in the editor as you build, in CI through the
GitHub Action, and over the wire at
POST /api/v1/check. Milliseconds, before any GPU is billed.
info
structure
R34
| Trigger | The model has an embedding with a numeric vocabSize and attention layers, but the final Linear feeding the Output projects to a different width. |
|---|---|
| Why | A language model's head must project back to vocab size (and is usually weight-tied to the embedding); a mismatch means the model cannot emit token logits. A classifier head over N classes is the expected exception, which is why this is info, not error. |
| Source | Weight-tying convention (Press & Wolf 2017); standard LM head contract. |
Why it is not a lint you can ignore
A structural mistake does not fail at review time and it does not fail at import
time. It fails when the module is constructed on the training node, after the job was queued and
the dataset was downloaded. That is why this runs before the spend and not after it.
Run this check on your own model
Free, no account needed
Every check
41 structural checks: 6 guardrail gates and 35 architecture advisor rules. See the full catalogue.
← R33 Deep attention stack without depth-scaled init ยท R35 KV cache exceeds serving budget at reference context →