Checks / performance
KV cache exceeds serving budget at reference context
Check R35. Runs in the editor as you build, in CI through the
GitHub Action, and over the wire at
POST /api/v1/check. Milliseconds, before any GPU is billed.
warn
performance
R35
| Trigger | Total fp16 K/V cache across all attention layers (GQA-aware: counts numKVHeads, skips MLA) exceeds 4 GB for a single 8,192-token sequence. |
|---|---|
| Why | That much cache for ONE sequence is gone before weights or activations load; on a 24 GB GPU it caps concurrency at a handful of requests. Raise the GQA ratio, switch to MLA, reduce depth/width, or accept a shorter serving context. |
| Source | Same per-token KV math as R31 and the serving calculator, applied as an absolute budget instead of a pattern. |
Why it is not a lint you can ignore
A structural mistake does not fail at review time and it does not fail at import
time. It fails when the module is constructed on the training node, after the job was queued and
the dataset was downloaded. That is why this runs before the spend and not after it.
Run this check on your own model
Free, no account needed
Every check
41 structural checks: 6 guardrail gates and 35 architecture advisor rules. See the full catalogue.