Models / t5

t5-base: GPU memory to fine-tune

Fine-tuning t5-base at batch 1 and 512 tokens takes about 4.76 GB of GPU memory for a full fine-tune (fits one T4 16 GB), and 1.05 GB with LoRA r=16 on a bf16 base (fits one T4 16 GB). Estimated from the model's structure, not measured. What fits on which card.

Reconstructed from its own config.json with no weights read. 4.4M downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure

223M

223,113,600 parameters

In the published checkpoint

223M

222,903,936 scalars · safetensors.total, read 2024-02-14

Delta

+0.09%

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32, 512 tokens over 50,000 samples (assumed; no dataset attached).

Layers

136

Will it forward-pass

Yes

Priced on

A10G (24GB)

Est. one run

$3.95

Memory: what fits on which card

Largest batch per card. Training is sized at 512 tokens per sample, serving at 512 tokens per sequence.

Full fine-tune, batch 1 at 512 tokens

4.76 GB

LoRA r=16, batch 1 at 512 tokens

1.05 GB

Serve: bf16 weights

0.45 GB

Serve: KV cache, 512 tokens

57 MB

Card	Full fine-tune	LoRA r=16	Serve bf16, 512 tokens
T4 16 GB	batch 20	batch 27	batch 237
L4 24 GB	batch 34	batch 42	batch 360
A100 40 GB	batch 63	batch 71	batch 606
A100 / H100 80 GB	batch 135	batch 143	batch 1220
H200 141 GB	batch 246	batch 254	batch 2157

Training: 512 tokens per sample (2048, or the model's context when shorter), AdamW with bf16 autocast, SDPA attention (no score matrix stored), 1.15x headroom on every card. Full fine-tune holds weights, gradients and optimizer state (1.2 x 16 bytes per parameter) plus activations. LoRA is r=16 on every linear layer but the head, base frozen in bf16 (2 bytes per parameter), adapters at full AdamW state, same activations. A two-card split is a contiguous layer cut (pipeline parallel), computed only where one card cannot hold batch 1. Serving: bf16 weights plus the KV cache of 512 tokens per sequence; prefill working set, CUDA context and framework reserve not counted.

These are estimates from the graph, not measurements. Against measured runs, held out by architecture, the estimate flagged 113 of the 131 jobs that did not fit their card and said "fits" on the other 18. Every measured run.

Structure

138 nodes. Output shapes are propagated from the input shape, batch dimension excluded, at 512 tokens: the config's maximum context.

	Layer	Type	Output shape
1	Input	Input	1 × 512
2	Encoder_Embedding	Embedding	1 × 512 × 768
3	Relative_Position_Bias	Relative Position Bias	1 × 512 × 768
4	Enc_Attn_1	Multi-Head Attention	1 × 512 × 768
5	Enc_Add_1	Add	1 × 512 × 768
6	Enc_Norm_1	LayerNorm	1 × 512 × 768
7	Enc_FFN_1	Feed Forward	1 × 512 × 768
8	Enc_Attn_2	Multi-Head Attention	1 × 512 × 768
9	Enc_Add_2	Add	1 × 512 × 768
10	Enc_Norm_2	LayerNorm	1 × 512 × 768
11	Enc_FFN_2	Feed Forward	1 × 512 × 768
12	Enc_Attn_3	Multi-Head Attention	1 × 512 × 768
13	Enc_Add_3	Add	1 × 512 × 768
14	Enc_Norm_3	LayerNorm	1 × 512 × 768
15	Enc_FFN_3	Feed Forward	1 × 512 × 768
16	Enc_Attn_4	Multi-Head Attention	1 × 512 × 768
17	Enc_Add_4	Add	1 × 512 × 768
18	Enc_Norm_4	LayerNorm	1 × 512 × 768
19	Enc_FFN_4	Feed Forward	1 × 512 × 768
20	Enc_Attn_5	Multi-Head Attention	1 × 512 × 768
21	Enc_Add_5	Add	1 × 512 × 768
22	Enc_Norm_5	LayerNorm	1 × 512 × 768
23	Enc_FFN_5	Feed Forward	1 × 512 × 768
24	Enc_Attn_6	Multi-Head Attention	1 × 512 × 768
25	Enc_Add_6	Add	1 × 512 × 768
26	Enc_Norm_6	LayerNorm	1 × 512 × 768
27	Enc_FFN_6	Feed Forward	1 × 512 × 768
28	Enc_Attn_7	Multi-Head Attention	1 × 512 × 768
29	Enc_Add_7	Add	1 × 512 × 768
30	Enc_Norm_7	LayerNorm	1 × 512 × 768
31	Enc_FFN_7	Feed Forward	1 × 512 × 768
32	Enc_Attn_8	Multi-Head Attention	1 × 512 × 768
33	Enc_Add_8	Add	1 × 512 × 768
34	Enc_Norm_8	LayerNorm	1 × 512 × 768
35	Enc_FFN_8	Feed Forward	1 × 512 × 768
36	Enc_Attn_9	Multi-Head Attention	1 × 512 × 768
37	Enc_Add_9	Add	1 × 512 × 768
38	Enc_Norm_9	LayerNorm	1 × 512 × 768
39	Enc_FFN_9	Feed Forward	1 × 512 × 768
40	Enc_Attn_10	Multi-Head Attention	1 × 512 × 768
41	Enc_Add_10	Add	1 × 512 × 768
42	Enc_Norm_10	LayerNorm	1 × 512 × 768
43	Enc_FFN_10	Feed Forward	1 × 512 × 768
44	Enc_Attn_11	Multi-Head Attention	1 × 512 × 768
45	Enc_Add_11	Add	1 × 512 × 768
46	Enc_Norm_11	LayerNorm	1 × 512 × 768
47	Enc_FFN_11	Feed Forward	1 × 512 × 768
48	Enc_Attn_12	Multi-Head Attention	1 × 512 × 768
49	Enc_Add_12	Add	1 × 512 × 768
50	Enc_Norm_12	LayerNorm	1 × 512 × 768
51	Enc_FFN_12	Feed Forward	1 × 512 × 768
52	Decoder_Embedding	Embedding	1 × 512 × 768
53	Dec_SelfAttn_1	Multi-Head Attention	1 × 512 × 768
54	Dec_Add_1	Add	1 × 512 × 768
55	Dec_Norm1_1	LayerNorm	1 × 512 × 768
56	Dec_CrossAttn_1	Cross-Attention	1 × 512 × 768
57	Dec_Add2_1	Add	1 × 512 × 768
58	Dec_Norm2_1	LayerNorm	1 × 512 × 768
59	Dec_FFN_1	Feed Forward	1 × 512 × 768
60	Dec_SelfAttn_2	Multi-Head Attention	1 × 512 × 768
61	Dec_Add_2	Add	1 × 512 × 768
62	Dec_Norm1_2	LayerNorm	1 × 512 × 768
63	Dec_CrossAttn_2	Cross-Attention	1 × 512 × 768
64	Dec_Add2_2	Add	1 × 512 × 768
65	Dec_Norm2_2	LayerNorm	1 × 512 × 768
66	Dec_FFN_2	Feed Forward	1 × 512 × 768
67	Dec_SelfAttn_3	Multi-Head Attention	1 × 512 × 768
68	Dec_Add_3	Add	1 × 512 × 768
69	Dec_Norm1_3	LayerNorm	1 × 512 × 768
70	Dec_CrossAttn_3	Cross-Attention	1 × 512 × 768
71	Dec_Add2_3	Add	1 × 512 × 768
72	Dec_Norm2_3	LayerNorm	1 × 512 × 768
73	Dec_FFN_3	Feed Forward	1 × 512 × 768
74	Dec_SelfAttn_4	Multi-Head Attention	1 × 512 × 768
75	Dec_Add_4	Add	1 × 512 × 768
76	Dec_Norm1_4	LayerNorm	1 × 512 × 768
77	Dec_CrossAttn_4	Cross-Attention	1 × 512 × 768
78	Dec_Add2_4	Add	1 × 512 × 768
79	Dec_Norm2_4	LayerNorm	1 × 512 × 768
80	Dec_FFN_4	Feed Forward	1 × 512 × 768
81	Dec_SelfAttn_5	Multi-Head Attention	1 × 512 × 768
82	Dec_Add_5	Add	1 × 512 × 768
83	Dec_Norm1_5	LayerNorm	1 × 512 × 768
84	Dec_CrossAttn_5	Cross-Attention	1 × 512 × 768
85	Dec_Add2_5	Add	1 × 512 × 768
86	Dec_Norm2_5	LayerNorm	1 × 512 × 768
87	Dec_FFN_5	Feed Forward	1 × 512 × 768
88	Dec_SelfAttn_6	Multi-Head Attention	1 × 512 × 768
89	Dec_Add_6	Add	1 × 512 × 768
90	Dec_Norm1_6	LayerNorm	1 × 512 × 768
91	Dec_CrossAttn_6	Cross-Attention	1 × 512 × 768
92	Dec_Add2_6	Add	1 × 512 × 768
93	Dec_Norm2_6	LayerNorm	1 × 512 × 768
94	Dec_FFN_6	Feed Forward	1 × 512 × 768
95	Dec_SelfAttn_7	Multi-Head Attention	1 × 512 × 768
96	Dec_Add_7	Add	1 × 512 × 768
97	Dec_Norm1_7	LayerNorm	1 × 512 × 768
98	Dec_CrossAttn_7	Cross-Attention	1 × 512 × 768
99	Dec_Add2_7	Add	1 × 512 × 768
100	Dec_Norm2_7	LayerNorm	1 × 512 × 768
101	Dec_FFN_7	Feed Forward	1 × 512 × 768
102	Dec_SelfAttn_8	Multi-Head Attention	1 × 512 × 768
103	Dec_Add_8	Add	1 × 512 × 768
104	Dec_Norm1_8	LayerNorm	1 × 512 × 768
105	Dec_CrossAttn_8	Cross-Attention	1 × 512 × 768
106	Dec_Add2_8	Add	1 × 512 × 768
107	Dec_Norm2_8	LayerNorm	1 × 512 × 768
108	Dec_FFN_8	Feed Forward	1 × 512 × 768
109	Dec_SelfAttn_9	Multi-Head Attention	1 × 512 × 768
110	Dec_Add_9	Add	1 × 512 × 768
111	Dec_Norm1_9	LayerNorm	1 × 512 × 768
112	Dec_CrossAttn_9	Cross-Attention	1 × 512 × 768
113	Dec_Add2_9	Add	1 × 512 × 768
114	Dec_Norm2_9	LayerNorm	1 × 512 × 768
115	Dec_FFN_9	Feed Forward	1 × 512 × 768
116	Dec_SelfAttn_10	Multi-Head Attention	1 × 512 × 768
117	Dec_Add_10	Add	1 × 512 × 768
118	Dec_Norm1_10	LayerNorm	1 × 512 × 768
119	Dec_CrossAttn_10	Cross-Attention	1 × 512 × 768
120	Dec_Add2_10	Add	1 × 512 × 768
121	Dec_Norm2_10	LayerNorm	1 × 512 × 768
122	Dec_FFN_10	Feed Forward	1 × 512 × 768
123	Dec_SelfAttn_11	Multi-Head Attention	1 × 512 × 768
124	Dec_Add_11	Add	1 × 512 × 768
125	Dec_Norm1_11	LayerNorm	1 × 512 × 768
126	Dec_CrossAttn_11	Cross-Attention	1 × 512 × 768
127	Dec_Add2_11	Add	1 × 512 × 768
128	Dec_Norm2_11	LayerNorm	1 × 512 × 768
129	Dec_FFN_11	Feed Forward	1 × 512 × 768
130	Dec_SelfAttn_12	Multi-Head Attention	1 × 512 × 768
131	Dec_Add_12	Add	1 × 512 × 768
132	Dec_Norm1_12	LayerNorm	1 × 512 × 768
133	Dec_CrossAttn_12	Cross-Attention	1 × 512 × 768
134	Dec_Add2_12	Add	1 × 512 × 768
135	Dec_Norm2_12	LayerNorm	1 × 512 × 768
136	Dec_FFN_12	Feed Forward	1 × 512 × 768
137	LM_Head	Linear	1 × 512 × 32128
138	Output	Output	1 × 512 × 32128

What the verifier says

infoAt 24 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init

What changed upstream

The change tracker follows google-t5/t5-base: every revision it has read, what changed in the config, tokenizer, chat template or weights between them, and an OSV advisory per change.

Revision history and changes Badge JSON OSV advisories All tracked changes

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace google-t5/t5-base --plan --share

Paste a model instead Markdown twin Index JSON POST /api/v1/plan

Other t5 checkpoints

t5-small

60.6M derived · +0.11% against the checkpoint