Models / whisper

whisper-large-v3-turbo: GPU memory to fine-tune

Fine-tuning whisper-large-v3-turbo at batch 1 and 512 tokens takes about 15.6 GB of GPU memory for a full fine-tune (borderline on one L4 24 GB), and 2.11 GB with LoRA r=16 on a bf16 base (fits one T4 16 GB). Estimated from the model's structure, not measured. What fits on which card.

Reconstructed from its own config.json with no weights read. 7.1M downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure

801M

801,484,800 parameters

In the published checkpoint

809M

808,878,080 scalars · safetensors.total, read 2026-09-06

Delta

-0.91%

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32, 512 tokens over 50,000 samples (assumed; no dataset attached).

Layers

121

Will it forward-pass

Yes

Priced on

A10G (24GB)

Est. one run

$2.54

Memory: what fits on which card

Largest batch per card. Training is sized at 512 tokens per sample, serving at 512 tokens per sequence.

Full fine-tune, batch 1 at 512 tokens

15.6 GB

LoRA r=16, batch 1 at 512 tokens

2.11 GB

Serve: bf16 weights

1.60 GB

Serve: KV cache, 512 tokens

0.10 GB

Card	Full fine-tune	LoRA r=16	Serve bf16, 512 tokens
T4 16 GB	2 cards at batch 1 (107 + 14 layers)	batch 51	batch 117
L4 24 GB	batch 23, borderline	batch 80	batch 183
A100 40 GB	batch 82	batch 139	batch 316
A100 / H100 80 GB	batch 230	batch 287	batch 648
H200 141 GB	batch 455	batch 513	batch 1154

Training: 512 tokens per sample (2048, or the model's context when shorter), AdamW with bf16 autocast, SDPA attention (no score matrix stored), 1.15x headroom on every card. Full fine-tune holds weights, gradients and optimizer state (1.2 x 16 bytes per parameter) plus activations. LoRA is r=16 on every linear layer but the head, base frozen in bf16 (2 bytes per parameter), adapters at full AdamW state, same activations. A two-card split is a contiguous layer cut (pipeline parallel), computed only where one card cannot hold batch 1. "Borderline" means the card holds batch 1 on the estimate but not on the conservative peak (the estimate x 1.43, at or above the measured peak on 95% of held-out runs): measure one step before reserving it. Serving: bf16 weights plus the KV cache of 512 tokens per sequence; prefill working set, CUDA context and framework reserve not counted.

These are estimates from the graph, not measurements. Against measured runs, held out by architecture, the estimate flagged 113 of the 131 jobs that did not fit their card and said "fits" on the other 18. Every measured run.

Structure

124 nodes. Output shapes are propagated from the input shape, batch dimension excluded, at 512 tokens: the config's maximum context.

	Layer	Type	Output shape
1	Input	Input	1 × 512
2	MelSpectrogram	Mel Spectrogram	1 × 80 × 1
3	Enc_Conv1	Conv1D	1 × 1280 × 1
4	Enc_Conv2	Conv1D	1 × 1280 × 1
5	Enc_ToTokens	Permute	1 × 1 × 1280
6	Enc_PosEnc	Positional Encoding	1 × 1 × 1280
7	Enc_Attn_1	Multi-Head Attention	1 × 1 × 1280
8	Enc_Norm_1	LayerNorm	1 × 1 × 1280
9	Enc_FFN_1	Feed Forward	1 × 1 × 1280
10	Enc_Attn_2	Multi-Head Attention	1 × 1 × 1280
11	Enc_Norm_2	LayerNorm	1 × 1 × 1280
12	Enc_FFN_2	Feed Forward	1 × 1 × 1280
13	Enc_Attn_3	Multi-Head Attention	1 × 1 × 1280
14	Enc_Norm_3	LayerNorm	1 × 1 × 1280
15	Enc_FFN_3	Feed Forward	1 × 1 × 1280
16	Enc_Attn_4	Multi-Head Attention	1 × 1 × 1280
17	Enc_Norm_4	LayerNorm	1 × 1 × 1280
18	Enc_FFN_4	Feed Forward	1 × 1 × 1280
19	Enc_Attn_5	Multi-Head Attention	1 × 1 × 1280
20	Enc_Norm_5	LayerNorm	1 × 1 × 1280
21	Enc_FFN_5	Feed Forward	1 × 1 × 1280
22	Enc_Attn_6	Multi-Head Attention	1 × 1 × 1280
23	Enc_Norm_6	LayerNorm	1 × 1 × 1280
24	Enc_FFN_6	Feed Forward	1 × 1 × 1280
25	Enc_Attn_7	Multi-Head Attention	1 × 1 × 1280
26	Enc_Norm_7	LayerNorm	1 × 1 × 1280
27	Enc_FFN_7	Feed Forward	1 × 1 × 1280
28	Enc_Attn_8	Multi-Head Attention	1 × 1 × 1280
29	Enc_Norm_8	LayerNorm	1 × 1 × 1280
30	Enc_FFN_8	Feed Forward	1 × 1 × 1280
31	Enc_Attn_9	Multi-Head Attention	1 × 1 × 1280
32	Enc_Norm_9	LayerNorm	1 × 1 × 1280
33	Enc_FFN_9	Feed Forward	1 × 1 × 1280
34	Enc_Attn_10	Multi-Head Attention	1 × 1 × 1280
35	Enc_Norm_10	LayerNorm	1 × 1 × 1280
36	Enc_FFN_10	Feed Forward	1 × 1 × 1280
37	Enc_Attn_11	Multi-Head Attention	1 × 1 × 1280
38	Enc_Norm_11	LayerNorm	1 × 1 × 1280
39	Enc_FFN_11	Feed Forward	1 × 1 × 1280
40	Enc_Attn_12	Multi-Head Attention	1 × 1 × 1280
41	Enc_Norm_12	LayerNorm	1 × 1 × 1280
42	Enc_FFN_12	Feed Forward	1 × 1 × 1280
43	Enc_Attn_13	Multi-Head Attention	1 × 1 × 1280
44	Enc_Norm_13	LayerNorm	1 × 1 × 1280
45	Enc_FFN_13	Feed Forward	1 × 1 × 1280
46	Enc_Attn_14	Multi-Head Attention	1 × 1 × 1280
47	Enc_Norm_14	LayerNorm	1 × 1 × 1280
48	Enc_FFN_14	Feed Forward	1 × 1 × 1280
49	Enc_Attn_15	Multi-Head Attention	1 × 1 × 1280
50	Enc_Norm_15	LayerNorm	1 × 1 × 1280
51	Enc_FFN_15	Feed Forward	1 × 1 × 1280
52	Enc_Attn_16	Multi-Head Attention	1 × 1 × 1280
53	Enc_Norm_16	LayerNorm	1 × 1 × 1280
54	Enc_FFN_16	Feed Forward	1 × 1 × 1280
55	Enc_Attn_17	Multi-Head Attention	1 × 1 × 1280
56	Enc_Norm_17	LayerNorm	1 × 1 × 1280
57	Enc_FFN_17	Feed Forward	1 × 1 × 1280
58	Enc_Attn_18	Multi-Head Attention	1 × 1 × 1280
59	Enc_Norm_18	LayerNorm	1 × 1 × 1280
60	Enc_FFN_18	Feed Forward	1 × 1 × 1280
61	Enc_Attn_19	Multi-Head Attention	1 × 1 × 1280
62	Enc_Norm_19	LayerNorm	1 × 1 × 1280
63	Enc_FFN_19	Feed Forward	1 × 1 × 1280
64	Enc_Attn_20	Multi-Head Attention	1 × 1 × 1280
65	Enc_Norm_20	LayerNorm	1 × 1 × 1280
66	Enc_FFN_20	Feed Forward	1 × 1 × 1280
67	Enc_Attn_21	Multi-Head Attention	1 × 1 × 1280
68	Enc_Norm_21	LayerNorm	1 × 1 × 1280
69	Enc_FFN_21	Feed Forward	1 × 1 × 1280
70	Enc_Attn_22	Multi-Head Attention	1 × 1 × 1280
71	Enc_Norm_22	LayerNorm	1 × 1 × 1280
72	Enc_FFN_22	Feed Forward	1 × 1 × 1280
73	Enc_Attn_23	Multi-Head Attention	1 × 1 × 1280
74	Enc_Norm_23	LayerNorm	1 × 1 × 1280
75	Enc_FFN_23	Feed Forward	1 × 1 × 1280
76	Enc_Attn_24	Multi-Head Attention	1 × 1 × 1280
77	Enc_Norm_24	LayerNorm	1 × 1 × 1280
78	Enc_FFN_24	Feed Forward	1 × 1 × 1280
79	Enc_Attn_25	Multi-Head Attention	1 × 1 × 1280
80	Enc_Norm_25	LayerNorm	1 × 1 × 1280
81	Enc_FFN_25	Feed Forward	1 × 1 × 1280
82	Enc_Attn_26	Multi-Head Attention	1 × 1 × 1280
83	Enc_Norm_26	LayerNorm	1 × 1 × 1280
84	Enc_FFN_26	Feed Forward	1 × 1 × 1280
85	Enc_Attn_27	Multi-Head Attention	1 × 1 × 1280
86	Enc_Norm_27	LayerNorm	1 × 1 × 1280
87	Enc_FFN_27	Feed Forward	1 × 1 × 1280
88	Enc_Attn_28	Multi-Head Attention	1 × 1 × 1280
89	Enc_Norm_28	LayerNorm	1 × 1 × 1280
90	Enc_FFN_28	Feed Forward	1 × 1 × 1280
91	Enc_Attn_29	Multi-Head Attention	1 × 1 × 1280
92	Enc_Norm_29	LayerNorm	1 × 1 × 1280
93	Enc_FFN_29	Feed Forward	1 × 1 × 1280
94	Enc_Attn_30	Multi-Head Attention	1 × 1 × 1280
95	Enc_Norm_30	LayerNorm	1 × 1 × 1280
96	Enc_FFN_30	Feed Forward	1 × 1 × 1280
97	Enc_Attn_31	Multi-Head Attention	1 × 1 × 1280
98	Enc_Norm_31	LayerNorm	1 × 1 × 1280
99	Enc_FFN_31	Feed Forward	1 × 1 × 1280
100	Enc_Attn_32	Multi-Head Attention	1 × 1 × 1280
101	Enc_Norm_32	LayerNorm	1 × 1 × 1280
102	Enc_FFN_32	Feed Forward	1 × 1 × 1280
103	decoder_tokens	Input	1 × 448
104	Dec_Token_Embedding	Embedding	1 × 448 × 1280
105	Dec_PosEmbedding	Learned Pos Embed	1 × 448 × 1280
106	Dec_SelfAttn_1	Causal Attention	1 × 448 × 1280
107	Dec_CrossAttn_1	Cross-Attention	1 × 448 × 1280
108	Dec_Norm_1	LayerNorm	1 × 448 × 1280
109	Dec_FFN_1	Feed Forward	1 × 448 × 1280
110	Dec_SelfAttn_2	Causal Attention	1 × 448 × 1280
111	Dec_CrossAttn_2	Cross-Attention	1 × 448 × 1280
112	Dec_Norm_2	LayerNorm	1 × 448 × 1280
113	Dec_FFN_2	Feed Forward	1 × 448 × 1280
114	Dec_SelfAttn_3	Causal Attention	1 × 448 × 1280
115	Dec_CrossAttn_3	Cross-Attention	1 × 448 × 1280
116	Dec_Norm_3	LayerNorm	1 × 448 × 1280
117	Dec_FFN_3	Feed Forward	1 × 448 × 1280
118	Dec_SelfAttn_4	Causal Attention	1 × 448 × 1280
119	Dec_CrossAttn_4	Cross-Attention	1 × 448 × 1280
120	Dec_Norm_4	LayerNorm	1 × 448 × 1280
121	Dec_FFN_4	Feed Forward	1 × 448 × 1280
122	Dec_Norm_Final	LayerNorm	1 × 448 × 1280
123	LM_Head	Linear	1 × 448 × 51866
124	Output	Output	1 × 448 × 51866

What the verifier says

infoAt 36 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init

What changed upstream

The change tracker follows openai/whisper-large-v3-turbo: every revision it has read, what changed in the config, tokenizer, chat template or weights between them, and an OSV advisory per change.

Revision history and changes Badge JSON OSV advisories All tracked changes

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace openai/whisper-large-v3-turbo --plan --share

Paste a model instead Markdown twin Index JSON POST /api/v1/plan

Other whisper checkpoints

Whisper-Hindi2Hinglish-Apex

801M derived · -0.91% against the checkpoint

Whisper-Hindi2Hinglish-Prime

1.54B derived · -0.49% against the checkpoint

whisper-large-v3

1.54B derived · -0.49% against the checkpoint