Models / deepseek_v3

DeepSeek-V3.1: GPU memory to fine-tune

Fine-tuning DeepSeek-V3.1 at batch 1 and 2048 tokens takes about 13.18 TB of GPU memory for a full fine-tune (at least 108 x H200 sharded), and 1.40 TB with LoRA r=16 on a bf16 base (at least 12 x H200 sharded). Estimated from the model's structure, not measured. What fits on which card.

Reconstructed from its own config.json with no weights read. 229K downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure

684B

684,490,096,128 parameters

In the published checkpoint

685B

684,531,386,000 scalars · safetensors.total, read 2025-09-05

Delta

-0.01%

quantized This checkpoint is stored quantized (fp8). The tensor count in the file counts stored elements under a packing scheme, not logical parameters, so the two numbers below are not measuring the same thing in either direction.

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32, 2048 tokens over 50,000 samples (assumed; no dataset attached).

Layers

383

Will it forward-pass

Yes

Priced on

A10G (24GB)

Est. one run

$3170.53

Memory: what fits on which card

Largest batch per card. Training is sized at 2048 tokens per sample, serving at 4096 tokens per sequence.

Full fine-tune, batch 1 at 2048 tokens

13.18 TB

LoRA r=16, batch 1 at 2048 tokens

1.40 TB

Serve: bf16 weights

1.37 TB

Serve: KV cache, 4096 tokens

0.29 GB

Card	Full fine-tune	LoRA r=16	Serve bf16, 4096 tokens
T4 16 GB	does not fit at batch 1	does not fit at batch 1	does not fit at batch 1
L4 24 GB	does not fit at batch 1	does not fit at batch 1	does not fit at batch 1
A100 40 GB	does not fit at batch 1	does not fit at batch 1	does not fit at batch 1
A100 / H100 80 GB	does not fit at batch 1	does not fit at batch 1	does not fit at batch 1
H200 141 GB	does not fit at batch 1	does not fit at batch 1	does not fit at batch 1
Sharded, by memory alone	at least 108 x H200 or 190 x 80 GB	at least 12 x H200 or 21 x 80 GB	at least 12 x H200 or 20 x 80 GB

Training: 2048 tokens per sample (2048, or the model's context when shorter; the structure table is at the config's maximum, 163840), AdamW with bf16 autocast, SDPA attention (no score matrix stored), 1.15x headroom on every card. Full fine-tune holds weights, gradients and optimizer state (1.2 x 16 bytes per parameter) plus activations. LoRA is r=16 on every linear layer but the head, base frozen in bf16 (2 bytes per parameter), adapters at full AdamW state, same activations. A two-card split is a contiguous layer cut (pipeline parallel), computed only where one card cannot hold batch 1. "Sharded" is a floor: the batch-1 peak with headroom divided across cards as if it split perfectly evenly (FSDP / ZeRO-3 for training, tensor parallel for serving), with no communication buffers counted. Serving: bf16 weights plus the KV cache of 4096 tokens per sequence; prefill working set, CUDA context and framework reserve not counted.

These are estimates from the graph, not measurements. Against measured runs, held out by architecture, the estimate flagged 113 of the 131 jobs that did not fit their card and said "fits" on the other 18. Every measured run.

Structure

386 nodes. Output shapes are propagated from the input shape, batch dimension excluded, at 163840 tokens: the config's maximum context, not the 2048 the memory above is sized at.

	Layer	Type	Output shape
1	Input	Input	1 × 163840
2	Embedding	Embedding	1 × 163840 × 7168
3	RoPE	RoPE	1 × 163840 × 7168
4	RMSNorm_1_1	RMSNorm	1 × 163840 × 7168
5	Attention_1	Multi-Head Latent Attention	1 × 163840 × 7168
6	Add_1_attn	Add	1 × 163840 × 7168
7	RMSNorm_1_2	RMSNorm	1 × 163840 × 7168
8	FFN_1	SwiGLU	1 × 163840 × 7168
9	Add_1_ffn	Add	1 × 163840 × 7168
10	RMSNorm_2_1	RMSNorm	1 × 163840 × 7168
11	Attention_2	Multi-Head Latent Attention	1 × 163840 × 7168
12	Add_2_attn	Add	1 × 163840 × 7168
13	RMSNorm_2_2	RMSNorm	1 × 163840 × 7168
14	FFN_2	SwiGLU	1 × 163840 × 7168
15	Add_2_ffn	Add	1 × 163840 × 7168
16	RMSNorm_3_1	RMSNorm	1 × 163840 × 7168
17	Attention_3	Multi-Head Latent Attention	1 × 163840 × 7168
18	Add_3_attn	Add	1 × 163840 × 7168
19	RMSNorm_3_2	RMSNorm	1 × 163840 × 7168
20	FFN_3	SwiGLU	1 × 163840 × 7168
21	Add_3_ffn	Add	1 × 163840 × 7168
22	RMSNorm_4_1	RMSNorm	1 × 163840 × 7168
23	Attention_4	Multi-Head Latent Attention	1 × 163840 × 7168
24	Add_4_attn	Add	1 × 163840 × 7168
25	RMSNorm_4_2	RMSNorm	1 × 163840 × 7168
26	MoE_4	Shared-Expert MoE	1 × 163840 × 7168
27	Add_4_ffn	Add	1 × 163840 × 7168
28	RMSNorm_5_1	RMSNorm	1 × 163840 × 7168
29	Attention_5	Multi-Head Latent Attention	1 × 163840 × 7168
30	Add_5_attn	Add	1 × 163840 × 7168
31	RMSNorm_5_2	RMSNorm	1 × 163840 × 7168
32	MoE_5	Shared-Expert MoE	1 × 163840 × 7168
33	Add_5_ffn	Add	1 × 163840 × 7168
34	RMSNorm_6_1	RMSNorm	1 × 163840 × 7168
35	Attention_6	Multi-Head Latent Attention	1 × 163840 × 7168
36	Add_6_attn	Add	1 × 163840 × 7168
37	RMSNorm_6_2	RMSNorm	1 × 163840 × 7168
38	MoE_6	Shared-Expert MoE	1 × 163840 × 7168
39	Add_6_ffn	Add	1 × 163840 × 7168
40	RMSNorm_7_1	RMSNorm	1 × 163840 × 7168
41	Attention_7	Multi-Head Latent Attention	1 × 163840 × 7168
42	Add_7_attn	Add	1 × 163840 × 7168
43	RMSNorm_7_2	RMSNorm	1 × 163840 × 7168
44	MoE_7	Shared-Expert MoE	1 × 163840 × 7168
45	Add_7_ffn	Add	1 × 163840 × 7168
46	RMSNorm_8_1	RMSNorm	1 × 163840 × 7168
47	Attention_8	Multi-Head Latent Attention	1 × 163840 × 7168
48	Add_8_attn	Add	1 × 163840 × 7168
49	RMSNorm_8_2	RMSNorm	1 × 163840 × 7168
50	MoE_8	Shared-Expert MoE	1 × 163840 × 7168
51	Add_8_ffn	Add	1 × 163840 × 7168
52	RMSNorm_9_1	RMSNorm	1 × 163840 × 7168
53	Attention_9	Multi-Head Latent Attention	1 × 163840 × 7168
54	Add_9_attn	Add	1 × 163840 × 7168
55	RMSNorm_9_2	RMSNorm	1 × 163840 × 7168
56	MoE_9	Shared-Expert MoE	1 × 163840 × 7168
57	Add_9_ffn	Add	1 × 163840 × 7168
58	RMSNorm_10_1	RMSNorm	1 × 163840 × 7168
59	Attention_10	Multi-Head Latent Attention	1 × 163840 × 7168
60	Add_10_attn	Add	1 × 163840 × 7168
61	RMSNorm_10_2	RMSNorm	1 × 163840 × 7168
62	MoE_10	Shared-Expert MoE	1 × 163840 × 7168
63	Add_10_ffn	Add	1 × 163840 × 7168
64	RMSNorm_11_1	RMSNorm	1 × 163840 × 7168
65	Attention_11	Multi-Head Latent Attention	1 × 163840 × 7168
66	Add_11_attn	Add	1 × 163840 × 7168
67	RMSNorm_11_2	RMSNorm	1 × 163840 × 7168
68	MoE_11	Shared-Expert MoE	1 × 163840 × 7168
69	Add_11_ffn	Add	1 × 163840 × 7168
70	RMSNorm_12_1	RMSNorm	1 × 163840 × 7168
71	Attention_12	Multi-Head Latent Attention	1 × 163840 × 7168
72	Add_12_attn	Add	1 × 163840 × 7168
73	RMSNorm_12_2	RMSNorm	1 × 163840 × 7168
74	MoE_12	Shared-Expert MoE	1 × 163840 × 7168
75	Add_12_ffn	Add	1 × 163840 × 7168
76	RMSNorm_13_1	RMSNorm	1 × 163840 × 7168
77	Attention_13	Multi-Head Latent Attention	1 × 163840 × 7168
78	Add_13_attn	Add	1 × 163840 × 7168
79	RMSNorm_13_2	RMSNorm	1 × 163840 × 7168
80	MoE_13	Shared-Expert MoE	1 × 163840 × 7168
81	Add_13_ffn	Add	1 × 163840 × 7168
82	RMSNorm_14_1	RMSNorm	1 × 163840 × 7168
83	Attention_14	Multi-Head Latent Attention	1 × 163840 × 7168
84	Add_14_attn	Add	1 × 163840 × 7168
85	RMSNorm_14_2	RMSNorm	1 × 163840 × 7168
86	MoE_14	Shared-Expert MoE	1 × 163840 × 7168
87	Add_14_ffn	Add	1 × 163840 × 7168
88	RMSNorm_15_1	RMSNorm	1 × 163840 × 7168
89	Attention_15	Multi-Head Latent Attention	1 × 163840 × 7168
90	Add_15_attn	Add	1 × 163840 × 7168
91	RMSNorm_15_2	RMSNorm	1 × 163840 × 7168
92	MoE_15	Shared-Expert MoE	1 × 163840 × 7168
93	Add_15_ffn	Add	1 × 163840 × 7168
94	RMSNorm_16_1	RMSNorm	1 × 163840 × 7168
95	Attention_16	Multi-Head Latent Attention	1 × 163840 × 7168
96	Add_16_attn	Add	1 × 163840 × 7168
97	RMSNorm_16_2	RMSNorm	1 × 163840 × 7168
98	MoE_16	Shared-Expert MoE	1 × 163840 × 7168
99	Add_16_ffn	Add	1 × 163840 × 7168
100	RMSNorm_17_1	RMSNorm	1 × 163840 × 7168
101	Attention_17	Multi-Head Latent Attention	1 × 163840 × 7168
102	Add_17_attn	Add	1 × 163840 × 7168
103	RMSNorm_17_2	RMSNorm	1 × 163840 × 7168
104	MoE_17	Shared-Expert MoE	1 × 163840 × 7168
105	Add_17_ffn	Add	1 × 163840 × 7168
106	RMSNorm_18_1	RMSNorm	1 × 163840 × 7168
107	Attention_18	Multi-Head Latent Attention	1 × 163840 × 7168
108	Add_18_attn	Add	1 × 163840 × 7168
109	RMSNorm_18_2	RMSNorm	1 × 163840 × 7168
110	MoE_18	Shared-Expert MoE	1 × 163840 × 7168
111	Add_18_ffn	Add	1 × 163840 × 7168
112	RMSNorm_19_1	RMSNorm	1 × 163840 × 7168
113	Attention_19	Multi-Head Latent Attention	1 × 163840 × 7168
114	Add_19_attn	Add	1 × 163840 × 7168
115	RMSNorm_19_2	RMSNorm	1 × 163840 × 7168
116	MoE_19	Shared-Expert MoE	1 × 163840 × 7168
117	Add_19_ffn	Add	1 × 163840 × 7168
118	RMSNorm_20_1	RMSNorm	1 × 163840 × 7168
119	Attention_20	Multi-Head Latent Attention	1 × 163840 × 7168
120	Add_20_attn	Add	1 × 163840 × 7168
121	RMSNorm_20_2	RMSNorm	1 × 163840 × 7168
122	MoE_20	Shared-Expert MoE	1 × 163840 × 7168
123	Add_20_ffn	Add	1 × 163840 × 7168
124	RMSNorm_21_1	RMSNorm	1 × 163840 × 7168
125	Attention_21	Multi-Head Latent Attention	1 × 163840 × 7168
126	Add_21_attn	Add	1 × 163840 × 7168
127	RMSNorm_21_2	RMSNorm	1 × 163840 × 7168
128	MoE_21	Shared-Expert MoE	1 × 163840 × 7168
129	Add_21_ffn	Add	1 × 163840 × 7168
130	RMSNorm_22_1	RMSNorm	1 × 163840 × 7168
131	Attention_22	Multi-Head Latent Attention	1 × 163840 × 7168
132	Add_22_attn	Add	1 × 163840 × 7168
133	RMSNorm_22_2	RMSNorm	1 × 163840 × 7168
134	MoE_22	Shared-Expert MoE	1 × 163840 × 7168
135	Add_22_ffn	Add	1 × 163840 × 7168
136	RMSNorm_23_1	RMSNorm	1 × 163840 × 7168
137	Attention_23	Multi-Head Latent Attention	1 × 163840 × 7168
138	Add_23_attn	Add	1 × 163840 × 7168
139	RMSNorm_23_2	RMSNorm	1 × 163840 × 7168
140	MoE_23	Shared-Expert MoE	1 × 163840 × 7168
141	Add_23_ffn	Add	1 × 163840 × 7168
142	RMSNorm_24_1	RMSNorm	1 × 163840 × 7168
143	Attention_24	Multi-Head Latent Attention	1 × 163840 × 7168
144	Add_24_attn	Add	1 × 163840 × 7168
145	RMSNorm_24_2	RMSNorm	1 × 163840 × 7168
146	MoE_24	Shared-Expert MoE	1 × 163840 × 7168
147	Add_24_ffn	Add	1 × 163840 × 7168
148	RMSNorm_25_1	RMSNorm	1 × 163840 × 7168
149	Attention_25	Multi-Head Latent Attention	1 × 163840 × 7168
150	Add_25_attn	Add	1 × 163840 × 7168
151	RMSNorm_25_2	RMSNorm	1 × 163840 × 7168
152	MoE_25	Shared-Expert MoE	1 × 163840 × 7168
153	Add_25_ffn	Add	1 × 163840 × 7168
154	RMSNorm_26_1	RMSNorm	1 × 163840 × 7168
155	Attention_26	Multi-Head Latent Attention	1 × 163840 × 7168
156	Add_26_attn	Add	1 × 163840 × 7168
157	RMSNorm_26_2	RMSNorm	1 × 163840 × 7168
158	MoE_26	Shared-Expert MoE	1 × 163840 × 7168
159	Add_26_ffn	Add	1 × 163840 × 7168
160	RMSNorm_27_1	RMSNorm	1 × 163840 × 7168
161	Attention_27	Multi-Head Latent Attention	1 × 163840 × 7168
162	Add_27_attn	Add	1 × 163840 × 7168
163	RMSNorm_27_2	RMSNorm	1 × 163840 × 7168
164	MoE_27	Shared-Expert MoE	1 × 163840 × 7168
165	Add_27_ffn	Add	1 × 163840 × 7168
166	RMSNorm_28_1	RMSNorm	1 × 163840 × 7168
167	Attention_28	Multi-Head Latent Attention	1 × 163840 × 7168
168	Add_28_attn	Add	1 × 163840 × 7168
169	RMSNorm_28_2	RMSNorm	1 × 163840 × 7168
170	MoE_28	Shared-Expert MoE	1 × 163840 × 7168
171	Add_28_ffn	Add	1 × 163840 × 7168
172	RMSNorm_29_1	RMSNorm	1 × 163840 × 7168
173	Attention_29	Multi-Head Latent Attention	1 × 163840 × 7168
174	Add_29_attn	Add	1 × 163840 × 7168
175	RMSNorm_29_2	RMSNorm	1 × 163840 × 7168
176	MoE_29	Shared-Expert MoE	1 × 163840 × 7168
177	Add_29_ffn	Add	1 × 163840 × 7168
178	RMSNorm_30_1	RMSNorm	1 × 163840 × 7168
179	Attention_30	Multi-Head Latent Attention	1 × 163840 × 7168
180	Add_30_attn	Add	1 × 163840 × 7168
181	RMSNorm_30_2	RMSNorm	1 × 163840 × 7168
182	MoE_30	Shared-Expert MoE	1 × 163840 × 7168
183	Add_30_ffn	Add	1 × 163840 × 7168
184	RMSNorm_31_1	RMSNorm	1 × 163840 × 7168
185	Attention_31	Multi-Head Latent Attention	1 × 163840 × 7168
186	Add_31_attn	Add	1 × 163840 × 7168
187	RMSNorm_31_2	RMSNorm	1 × 163840 × 7168
188	MoE_31	Shared-Expert MoE	1 × 163840 × 7168
189	Add_31_ffn	Add	1 × 163840 × 7168
190	RMSNorm_32_1	RMSNorm	1 × 163840 × 7168
191	Attention_32	Multi-Head Latent Attention	1 × 163840 × 7168
192	Add_32_attn	Add	1 × 163840 × 7168
193	RMSNorm_32_2	RMSNorm	1 × 163840 × 7168
194	MoE_32	Shared-Expert MoE	1 × 163840 × 7168
195	Add_32_ffn	Add	1 × 163840 × 7168
196	RMSNorm_33_1	RMSNorm	1 × 163840 × 7168
197	Attention_33	Multi-Head Latent Attention	1 × 163840 × 7168
198	Add_33_attn	Add	1 × 163840 × 7168
199	RMSNorm_33_2	RMSNorm	1 × 163840 × 7168
200	MoE_33	Shared-Expert MoE	1 × 163840 × 7168
201	Add_33_ffn	Add	1 × 163840 × 7168
202	RMSNorm_34_1	RMSNorm	1 × 163840 × 7168
203	Attention_34	Multi-Head Latent Attention	1 × 163840 × 7168
204	Add_34_attn	Add	1 × 163840 × 7168
205	RMSNorm_34_2	RMSNorm	1 × 163840 × 7168
206	MoE_34	Shared-Expert MoE	1 × 163840 × 7168
207	Add_34_ffn	Add	1 × 163840 × 7168
208	RMSNorm_35_1	RMSNorm	1 × 163840 × 7168
209	Attention_35	Multi-Head Latent Attention	1 × 163840 × 7168
210	Add_35_attn	Add	1 × 163840 × 7168
211	RMSNorm_35_2	RMSNorm	1 × 163840 × 7168
212	MoE_35	Shared-Expert MoE	1 × 163840 × 7168
213	Add_35_ffn	Add	1 × 163840 × 7168
214	RMSNorm_36_1	RMSNorm	1 × 163840 × 7168
215	Attention_36	Multi-Head Latent Attention	1 × 163840 × 7168
216	Add_36_attn	Add	1 × 163840 × 7168
217	RMSNorm_36_2	RMSNorm	1 × 163840 × 7168
218	MoE_36	Shared-Expert MoE	1 × 163840 × 7168
219	Add_36_ffn	Add	1 × 163840 × 7168
220	RMSNorm_37_1	RMSNorm	1 × 163840 × 7168
221	Attention_37	Multi-Head Latent Attention	1 × 163840 × 7168
222	Add_37_attn	Add	1 × 163840 × 7168
223	RMSNorm_37_2	RMSNorm	1 × 163840 × 7168
224	MoE_37	Shared-Expert MoE	1 × 163840 × 7168
225	Add_37_ffn	Add	1 × 163840 × 7168
226	RMSNorm_38_1	RMSNorm	1 × 163840 × 7168
227	Attention_38	Multi-Head Latent Attention	1 × 163840 × 7168
228	Add_38_attn	Add	1 × 163840 × 7168
229	RMSNorm_38_2	RMSNorm	1 × 163840 × 7168
230	MoE_38	Shared-Expert MoE	1 × 163840 × 7168
231	Add_38_ffn	Add	1 × 163840 × 7168
232	RMSNorm_39_1	RMSNorm	1 × 163840 × 7168
233	Attention_39	Multi-Head Latent Attention	1 × 163840 × 7168
234	Add_39_attn	Add	1 × 163840 × 7168
235	RMSNorm_39_2	RMSNorm	1 × 163840 × 7168
236	MoE_39	Shared-Expert MoE	1 × 163840 × 7168
237	Add_39_ffn	Add	1 × 163840 × 7168
238	RMSNorm_40_1	RMSNorm	1 × 163840 × 7168
239	Attention_40	Multi-Head Latent Attention	1 × 163840 × 7168
240	Add_40_attn	Add	1 × 163840 × 7168
241	RMSNorm_40_2	RMSNorm	1 × 163840 × 7168
242	MoE_40	Shared-Expert MoE	1 × 163840 × 7168
243	Add_40_ffn	Add	1 × 163840 × 7168
244	RMSNorm_41_1	RMSNorm	1 × 163840 × 7168
245	Attention_41	Multi-Head Latent Attention	1 × 163840 × 7168
246	Add_41_attn	Add	1 × 163840 × 7168
247	RMSNorm_41_2	RMSNorm	1 × 163840 × 7168
248	MoE_41	Shared-Expert MoE	1 × 163840 × 7168
249	Add_41_ffn	Add	1 × 163840 × 7168
250	RMSNorm_42_1	RMSNorm	1 × 163840 × 7168
251	Attention_42	Multi-Head Latent Attention	1 × 163840 × 7168
252	Add_42_attn	Add	1 × 163840 × 7168
253	RMSNorm_42_2	RMSNorm	1 × 163840 × 7168
254	MoE_42	Shared-Expert MoE	1 × 163840 × 7168
255	Add_42_ffn	Add	1 × 163840 × 7168
256	RMSNorm_43_1	RMSNorm	1 × 163840 × 7168
257	Attention_43	Multi-Head Latent Attention	1 × 163840 × 7168
258	Add_43_attn	Add	1 × 163840 × 7168
259	RMSNorm_43_2	RMSNorm	1 × 163840 × 7168
260	MoE_43	Shared-Expert MoE	1 × 163840 × 7168
261	Add_43_ffn	Add	1 × 163840 × 7168
262	RMSNorm_44_1	RMSNorm	1 × 163840 × 7168
263	Attention_44	Multi-Head Latent Attention	1 × 163840 × 7168
264	Add_44_attn	Add	1 × 163840 × 7168
265	RMSNorm_44_2	RMSNorm	1 × 163840 × 7168
266	MoE_44	Shared-Expert MoE	1 × 163840 × 7168
267	Add_44_ffn	Add	1 × 163840 × 7168
268	RMSNorm_45_1	RMSNorm	1 × 163840 × 7168
269	Attention_45	Multi-Head Latent Attention	1 × 163840 × 7168
270	Add_45_attn	Add	1 × 163840 × 7168
271	RMSNorm_45_2	RMSNorm	1 × 163840 × 7168
272	MoE_45	Shared-Expert MoE	1 × 163840 × 7168
273	Add_45_ffn	Add	1 × 163840 × 7168
274	RMSNorm_46_1	RMSNorm	1 × 163840 × 7168
275	Attention_46	Multi-Head Latent Attention	1 × 163840 × 7168
276	Add_46_attn	Add	1 × 163840 × 7168
277	RMSNorm_46_2	RMSNorm	1 × 163840 × 7168
278	MoE_46	Shared-Expert MoE	1 × 163840 × 7168
279	Add_46_ffn	Add	1 × 163840 × 7168
280	RMSNorm_47_1	RMSNorm	1 × 163840 × 7168
281	Attention_47	Multi-Head Latent Attention	1 × 163840 × 7168
282	Add_47_attn	Add	1 × 163840 × 7168
283	RMSNorm_47_2	RMSNorm	1 × 163840 × 7168
284	MoE_47	Shared-Expert MoE	1 × 163840 × 7168
285	Add_47_ffn	Add	1 × 163840 × 7168
286	RMSNorm_48_1	RMSNorm	1 × 163840 × 7168
287	Attention_48	Multi-Head Latent Attention	1 × 163840 × 7168
288	Add_48_attn	Add	1 × 163840 × 7168
289	RMSNorm_48_2	RMSNorm	1 × 163840 × 7168
290	MoE_48	Shared-Expert MoE	1 × 163840 × 7168
291	Add_48_ffn	Add	1 × 163840 × 7168
292	RMSNorm_49_1	RMSNorm	1 × 163840 × 7168
293	Attention_49	Multi-Head Latent Attention	1 × 163840 × 7168
294	Add_49_attn	Add	1 × 163840 × 7168
295	RMSNorm_49_2	RMSNorm	1 × 163840 × 7168
296	MoE_49	Shared-Expert MoE	1 × 163840 × 7168
297	Add_49_ffn	Add	1 × 163840 × 7168
298	RMSNorm_50_1	RMSNorm	1 × 163840 × 7168
299	Attention_50	Multi-Head Latent Attention	1 × 163840 × 7168
300	Add_50_attn	Add	1 × 163840 × 7168
301	RMSNorm_50_2	RMSNorm	1 × 163840 × 7168
302	MoE_50	Shared-Expert MoE	1 × 163840 × 7168
303	Add_50_ffn	Add	1 × 163840 × 7168
304	RMSNorm_51_1	RMSNorm	1 × 163840 × 7168
305	Attention_51	Multi-Head Latent Attention	1 × 163840 × 7168
306	Add_51_attn	Add	1 × 163840 × 7168
307	RMSNorm_51_2	RMSNorm	1 × 163840 × 7168
308	MoE_51	Shared-Expert MoE	1 × 163840 × 7168
309	Add_51_ffn	Add	1 × 163840 × 7168
310	RMSNorm_52_1	RMSNorm	1 × 163840 × 7168
311	Attention_52	Multi-Head Latent Attention	1 × 163840 × 7168
312	Add_52_attn	Add	1 × 163840 × 7168
313	RMSNorm_52_2	RMSNorm	1 × 163840 × 7168
314	MoE_52	Shared-Expert MoE	1 × 163840 × 7168
315	Add_52_ffn	Add	1 × 163840 × 7168
316	RMSNorm_53_1	RMSNorm	1 × 163840 × 7168
317	Attention_53	Multi-Head Latent Attention	1 × 163840 × 7168
318	Add_53_attn	Add	1 × 163840 × 7168
319	RMSNorm_53_2	RMSNorm	1 × 163840 × 7168
320	MoE_53	Shared-Expert MoE	1 × 163840 × 7168
321	Add_53_ffn	Add	1 × 163840 × 7168
322	RMSNorm_54_1	RMSNorm	1 × 163840 × 7168
323	Attention_54	Multi-Head Latent Attention	1 × 163840 × 7168
324	Add_54_attn	Add	1 × 163840 × 7168
325	RMSNorm_54_2	RMSNorm	1 × 163840 × 7168
326	MoE_54	Shared-Expert MoE	1 × 163840 × 7168
327	Add_54_ffn	Add	1 × 163840 × 7168
328	RMSNorm_55_1	RMSNorm	1 × 163840 × 7168
329	Attention_55	Multi-Head Latent Attention	1 × 163840 × 7168
330	Add_55_attn	Add	1 × 163840 × 7168
331	RMSNorm_55_2	RMSNorm	1 × 163840 × 7168
332	MoE_55	Shared-Expert MoE	1 × 163840 × 7168
333	Add_55_ffn	Add	1 × 163840 × 7168
334	RMSNorm_56_1	RMSNorm	1 × 163840 × 7168
335	Attention_56	Multi-Head Latent Attention	1 × 163840 × 7168
336	Add_56_attn	Add	1 × 163840 × 7168
337	RMSNorm_56_2	RMSNorm	1 × 163840 × 7168
338	MoE_56	Shared-Expert MoE	1 × 163840 × 7168
339	Add_56_ffn	Add	1 × 163840 × 7168
340	RMSNorm_57_1	RMSNorm	1 × 163840 × 7168
341	Attention_57	Multi-Head Latent Attention	1 × 163840 × 7168
342	Add_57_attn	Add	1 × 163840 × 7168
343	RMSNorm_57_2	RMSNorm	1 × 163840 × 7168
344	MoE_57	Shared-Expert MoE	1 × 163840 × 7168
345	Add_57_ffn	Add	1 × 163840 × 7168
346	RMSNorm_58_1	RMSNorm	1 × 163840 × 7168
347	Attention_58	Multi-Head Latent Attention	1 × 163840 × 7168
348	Add_58_attn	Add	1 × 163840 × 7168
349	RMSNorm_58_2	RMSNorm	1 × 163840 × 7168
350	MoE_58	Shared-Expert MoE	1 × 163840 × 7168
351	Add_58_ffn	Add	1 × 163840 × 7168
352	RMSNorm_59_1	RMSNorm	1 × 163840 × 7168
353	Attention_59	Multi-Head Latent Attention	1 × 163840 × 7168
354	Add_59_attn	Add	1 × 163840 × 7168
355	RMSNorm_59_2	RMSNorm	1 × 163840 × 7168
356	MoE_59	Shared-Expert MoE	1 × 163840 × 7168
357	Add_59_ffn	Add	1 × 163840 × 7168
358	RMSNorm_60_1	RMSNorm	1 × 163840 × 7168
359	Attention_60	Multi-Head Latent Attention	1 × 163840 × 7168
360	Add_60_attn	Add	1 × 163840 × 7168
361	RMSNorm_60_2	RMSNorm	1 × 163840 × 7168
362	MoE_60	Shared-Expert MoE	1 × 163840 × 7168
363	Add_60_ffn	Add	1 × 163840 × 7168
364	RMSNorm_61_1	RMSNorm	1 × 163840 × 7168
365	Attention_61	Multi-Head Latent Attention	1 × 163840 × 7168
366	Add_61_attn	Add	1 × 163840 × 7168
367	RMSNorm_61_2	RMSNorm	1 × 163840 × 7168
368	MoE_61	Shared-Expert MoE	1 × 163840 × 7168
369	Add_61_ffn	Add	1 × 163840 × 7168
370	MTP_1_HiddenNorm	RMSNorm	1 × 163840 × 7168
371	MTP_1_Embedding	Embedding	1 × 163840 × 7168
372	MTP_1_EmbeddingNorm	RMSNorm	1 × 163840 × 7168
373	MTP_1_Concat	Concatenate	1 × 163840 × 14336
374	MTP_1_Projection	Linear	1 × 163840 × 7168
375	RMSNorm_MTP_1_1	RMSNorm	1 × 163840 × 7168
376	Attention_MTP_1	Multi-Head Latent Attention	1 × 163840 × 7168
377	Add_MTP_1_attn	Add	1 × 163840 × 7168
378	RMSNorm_MTP_1_2	RMSNorm	1 × 163840 × 7168
379	MoE_MTP_1	Shared-Expert MoE	1 × 163840 × 7168
380	Add_MTP_1_ffn	Add	1 × 163840 × 7168
381	MTP_1_Norm	RMSNorm	1 × 163840 × 7168
382	MTP_1_Head	Linear	1 × 163840 × 129280
383	Final_RMSNorm	RMSNorm	1 × 163840 × 7168
384	LM_Head	Linear	1 × 163840 × 129280
385	Output	Output	1 × 163840 × 129280
386	MTP_1_Output	Output	1 × 163840 × 129280

What the verifier says

infoAt 62 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init

What changed upstream

The change tracker follows deepseek-ai/DeepSeek-V3.1: every revision it has read, what changed in the config, tokenizer, chat template or weights between them, and an OSV advisory per change.

Revision history and changes Badge JSON OSV advisories All tracked changes

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace deepseek-ai/DeepSeek-V3.1 --plan --share

Paste a model instead Markdown twin Index JSON POST /api/v1/plan

Other deepseek_v3 checkpoints

DeepSeek-R1

684B derived · +0.00% against the checkpoint

DeepSeek-R1-0528

684B derived · -0.01% against the checkpoint

DeepSeek-R1-MXFP4

684B derived · +84.6% against the checkpoint

DeepSeek-V3

684B derived · -0.01% against the checkpoint