Data / Memory calibration

How close is the training-peak estimate, measured

Before a job takes a reservation, the plan says its peak training memory and the smallest card that holds it. This page puts that number next to accelerate estimate-memory and next to what the GPU actually allocated, for every run we measured. The rows are below and in memory-calibration.json.

Each run: three AdamW steps, fp32 weights under bf16 autocast, torch.cuda.max_memory_allocated, torch 2.8, on Modal. A10G and H100 at batch 8, 32 and 96; then 76 runs on a 24 GB L4 at batches chosen so an estimate lands near the card, which puts some over it on purpose. A run that ran out of memory is recorded as that: no peak, and "does not fit" for that card and every smaller one, never for a larger one. Measured 2026-10-17. Every number on this page is computed from those rows and from the factors the product ships, not typed by hand.

Predicted against measured

accelerate reads parameter bytes and nothing else: model, gradients and two Adam states. It counts no activations, so it sits under the line on every run where activations matter, which is most of them. That is a property of what it reads, not a mistake in its arithmetic. The plan reads the graph, so it can add every layer's output at your batch.

Four ways to say it

Card verdict: for each run and each of 16, 24, 40, 80 GB, does the method say the run fits (the plan with 15% headroom, accelerate as printed), against whether the measured peak fit. Do not read the last column as our edge. Most of these runs are small enough that any estimate says "fits", and accelerate never says anything else on a run that fit, so on those it cannot be wrong. The whole gap in the last column is the did-not-fit checks. The columns that separate the methods are the fifth and sixth: of the 131 times a run did not fit a card, how often the method said it would (the mistake that ends in an out-of-memory error after the reservation is taken), and how often it refused a run that fit (the mistake that buys a bigger card than the job needed). We make the second one 19 times held out; 7 of them are at the edge of the card (the run's measured peak is within 15% of it, inside the headroom the plan keeps), and the rest are over-estimates, the largest on the U-Net and the recurrent runs.

Every time a run did not fit a card

With the upper bound. The plan also carries a conservative peak: the estimate times an upper multiplier read from these rows alone, held out by architecture. Over the 91 fitted runs of 1 GB or more, measured peak over held-out prediction is at most x1.17 (90% of held-out runs) and x1.43 (95% of held-out runs); the plan ships the 95% one, and no bound under 1 GB (allocator overhead dominates there, so a job's bound is never under 1 GB). Each card is then fits (the bound with 15% headroom fits), borderline (the estimate is inside the card, the bound with headroom is not: measure once) or does not fit (the estimate alone is past the card). Of the 131 did-not-fit checks, the upper bound would have flagged 131: 29 as borderline and 102 as does not fit, against 113 flagged by the estimate alone. At x1.17 (90% of held-out runs), 129 (27 borderline, 102 does not fit). The price is on the runs that fit: of 571 such checks, 27 are borderline at the 95% bound (14 at 90%) and 13 are called does not fit. On the 25 runs measured after the fit, with the shipped factors, the 95% bound calls 0 of 42 did-not-fit checks "fits" (the estimate alone: 12), and 6 of the 6 runs that fit peaked at or under it.

How the numbers are made

Neurarch. Peak = 1.20 x 16 bytes per parameter (weights, gradients, two Adam states) + 0.88 x every layer's bf16 output at the batch + 1.34 x the intermediates inside each layer (every weight matrix's output per token, less the layer's own output: a feed-forward block's hidden activation, an attention block's q, k, v and context) + 3.64 x the attention scores eager attention stores for backward + 5.20 x a recurrent cell's gate states for every step + 2.02 x the largest single tensor, whose gradient and fp32 copy exist next to everything saved at the peak, all in bf16 elements and all read from the typed graph. The factors are fitted on relative error over the 178 runs not measured to check them, with every out-of-memory run as an inequality: the prediction times 1.15 must reach the card it ran out on, and a shortfall is penalised. So the "shipped factors" row is in-sample. The "held out" row refits all six with each architecture left out entirely, then predicts it. Runs under 1 GB weigh 0.5 in the fit because allocator overhead dominates them, and are left out of the error columns. The score term is counted only on layers the exported code ran eagerly (nn.MultiheadAttention returning its weights); GQA and MLA export through scaled_dot_product_attention and store none. The plan adds 15% headroom before choosing a card.

accelerate. The "Training using Adam" column of accelerate estimate-memory, reproduced from its source (accelerate 1.10.1, estimate_training_usage): the largest of model, gradients, optimizer and step sizes, from parameter bytes. That is 16 bytes a parameter in float32 and 8 in bf16 mixed precision.

Out of sample, two ways. Held out by architecture is the "held out" row above. Held out by regime: the six factors fitted on the A10G and H100 runs alone, then used on the 51 L4 runs, which sit at the edge of a 24 GB card at batches up to the thousands: a median error of 9.9% on the 15 that did not run out of memory (13 within 25%), 77 of 79 did-not-fit card checks flagged, 8 runs that fit called "does not fit". And 25 more L4 runs, at architectures and batches no earlier row has, were measured after the shipped factors were fixed (marked "after the fit"; never in any fit): there the shipped factors are off by a median 14.0% on the 6 that fit (5 within 25%), flagged 30 of 42 did-not-fit card checks and called 4 that fit "does not fit". A name with "cut from" is a reduced variant of a template whose weights alone are past any single card: the same layer types and wiring at smaller depth, vocabulary or sequence length (scripts/memory-calibration/variantGraphs.ts).

Known misses. The U-Net's skip concatenations are counted as new tensors on top of the ones they join: held out, 51% over. Recurrent cells rest on three templates, so the gate-state factor is the least supported of the six: held out, the recurrent runs above 1 GB miss between 75% under and 287% over. Under 1 GB, allocator overhead dominates. Of the 18 did-not-fit checks still called "fits", 17 are at the line: the prediction with headroom lands within 20% under the card, the run needed more than that, and the out-of-memory error is often a single multi-GB allocation (a score matrix, or the logits in fp32) that a fragmented cache cannot place. The other 1: cnn-lstm-1d.

Not counted. 9 runs are listed below and kept out of every fit and every number above, because the model that ran is not the graph the plan reads: graph-sage-rec, the exported example input ([4, 128], [2, 4]) has no batch axis, so the run measured one fixed graph whatever the batch, and the per-sample features do not describe it; lightgcn, the exported example input ([4], [4], [2, 4]) has no batch axis, so the run measured one fixed graph whatever the batch, and the per-sample features do not describe it; mixtral-block, measured on an export whose experts had two matrices where the graph has three (0.98B parameters built against 1.45B planned); the exporter now builds three, and the corrected model is past a 24 GB card on weights alone.

All 212 runs