Models / minimax_m2

MiniMax-M2

Reconstructed from its own config.json with no weights read. 221K downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure

229B

228,689,504,640 parameters

In the published checkpoint

229B

228,689,764,864 scalars · safetensors.total, read 2025-12-23

Delta

-0.00%

quantized This checkpoint is stored quantized (fp8). The tensor count in the file counts stored elements under a packing scheme, not logical parameters, so the two numbers below are not measuring the same thing in either direction.

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32 over 50,000 samples (assumed; no dataset attached). GPU fit is fp32 weights plus gradients plus two Adam moments (16 bytes per parameter) with 1.3x headroom; activations are not included and grow with batch size.

Layers

376

Will it forward-pass

Yes

Priced on

A10G (24GB)

Est. one run

$627749.17

Card	Memory
T4 (16GB)	weights + activations	does not fit
A100 (40GB)	weights + activations	does not fit
H100 (80GB)	weights + activations	does not fit

Structure

378 nodes. Output shapes are propagated from the input shape, batch dimension excluded.

	Layer	Type	Output shape
1	Input	Input	1 × 196608
2	Embedding	Embedding	1 × 196608 × 3072
3	RoPE	RoPE	1 × 196608 × 3072
4	RMSNorm_1_1	RMSNorm	1 × 196608 × 3072
5	Attention_1	Grouped Query Attn	1 × 196608 × 3072
6	Add_1_attn	Add	1 × 196608 × 3072
7	RMSNorm_1_2	RMSNorm	1 × 196608 × 3072
8	MoE_1	MoE Layer	1 × 196608 × 3072
9	Add_1_ffn	Add	1 × 196608 × 3072
10	RMSNorm_2_1	RMSNorm	1 × 196608 × 3072
11	Attention_2	Grouped Query Attn	1 × 196608 × 3072
12	Add_2_attn	Add	1 × 196608 × 3072
13	RMSNorm_2_2	RMSNorm	1 × 196608 × 3072
14	MoE_2	MoE Layer	1 × 196608 × 3072
15	Add_2_ffn	Add	1 × 196608 × 3072
16	RMSNorm_3_1	RMSNorm	1 × 196608 × 3072
17	Attention_3	Grouped Query Attn	1 × 196608 × 3072
18	Add_3_attn	Add	1 × 196608 × 3072
19	RMSNorm_3_2	RMSNorm	1 × 196608 × 3072
20	MoE_3	MoE Layer	1 × 196608 × 3072
21	Add_3_ffn	Add	1 × 196608 × 3072
22	RMSNorm_4_1	RMSNorm	1 × 196608 × 3072
23	Attention_4	Grouped Query Attn	1 × 196608 × 3072
24	Add_4_attn	Add	1 × 196608 × 3072
25	RMSNorm_4_2	RMSNorm	1 × 196608 × 3072
26	MoE_4	MoE Layer	1 × 196608 × 3072
27	Add_4_ffn	Add	1 × 196608 × 3072
28	RMSNorm_5_1	RMSNorm	1 × 196608 × 3072
29	Attention_5	Grouped Query Attn	1 × 196608 × 3072
30	Add_5_attn	Add	1 × 196608 × 3072
31	RMSNorm_5_2	RMSNorm	1 × 196608 × 3072
32	MoE_5	MoE Layer	1 × 196608 × 3072
33	Add_5_ffn	Add	1 × 196608 × 3072
34	RMSNorm_6_1	RMSNorm	1 × 196608 × 3072
35	Attention_6	Grouped Query Attn	1 × 196608 × 3072
36	Add_6_attn	Add	1 × 196608 × 3072
37	RMSNorm_6_2	RMSNorm	1 × 196608 × 3072
38	MoE_6	MoE Layer	1 × 196608 × 3072
39	Add_6_ffn	Add	1 × 196608 × 3072
40	RMSNorm_7_1	RMSNorm	1 × 196608 × 3072
41	Attention_7	Grouped Query Attn	1 × 196608 × 3072
42	Add_7_attn	Add	1 × 196608 × 3072
43	RMSNorm_7_2	RMSNorm	1 × 196608 × 3072
44	MoE_7	MoE Layer	1 × 196608 × 3072
45	Add_7_ffn	Add	1 × 196608 × 3072
46	RMSNorm_8_1	RMSNorm	1 × 196608 × 3072
47	Attention_8	Grouped Query Attn	1 × 196608 × 3072
48	Add_8_attn	Add	1 × 196608 × 3072
49	RMSNorm_8_2	RMSNorm	1 × 196608 × 3072
50	MoE_8	MoE Layer	1 × 196608 × 3072
51	Add_8_ffn	Add	1 × 196608 × 3072
52	RMSNorm_9_1	RMSNorm	1 × 196608 × 3072
53	Attention_9	Grouped Query Attn	1 × 196608 × 3072
54	Add_9_attn	Add	1 × 196608 × 3072
55	RMSNorm_9_2	RMSNorm	1 × 196608 × 3072
56	MoE_9	MoE Layer	1 × 196608 × 3072
57	Add_9_ffn	Add	1 × 196608 × 3072
58	RMSNorm_10_1	RMSNorm	1 × 196608 × 3072
59	Attention_10	Grouped Query Attn	1 × 196608 × 3072
60	Add_10_attn	Add	1 × 196608 × 3072
61	RMSNorm_10_2	RMSNorm	1 × 196608 × 3072
62	MoE_10	MoE Layer	1 × 196608 × 3072
63	Add_10_ffn	Add	1 × 196608 × 3072
64	RMSNorm_11_1	RMSNorm	1 × 196608 × 3072
65	Attention_11	Grouped Query Attn	1 × 196608 × 3072
66	Add_11_attn	Add	1 × 196608 × 3072
67	RMSNorm_11_2	RMSNorm	1 × 196608 × 3072
68	MoE_11	MoE Layer	1 × 196608 × 3072
69	Add_11_ffn	Add	1 × 196608 × 3072
70	RMSNorm_12_1	RMSNorm	1 × 196608 × 3072
71	Attention_12	Grouped Query Attn	1 × 196608 × 3072
72	Add_12_attn	Add	1 × 196608 × 3072
73	RMSNorm_12_2	RMSNorm	1 × 196608 × 3072
74	MoE_12	MoE Layer	1 × 196608 × 3072
75	Add_12_ffn	Add	1 × 196608 × 3072
76	RMSNorm_13_1	RMSNorm	1 × 196608 × 3072
77	Attention_13	Grouped Query Attn	1 × 196608 × 3072
78	Add_13_attn	Add	1 × 196608 × 3072
79	RMSNorm_13_2	RMSNorm	1 × 196608 × 3072
80	MoE_13	MoE Layer	1 × 196608 × 3072
81	Add_13_ffn	Add	1 × 196608 × 3072
82	RMSNorm_14_1	RMSNorm	1 × 196608 × 3072
83	Attention_14	Grouped Query Attn	1 × 196608 × 3072
84	Add_14_attn	Add	1 × 196608 × 3072
85	RMSNorm_14_2	RMSNorm	1 × 196608 × 3072
86	MoE_14	MoE Layer	1 × 196608 × 3072
87	Add_14_ffn	Add	1 × 196608 × 3072
88	RMSNorm_15_1	RMSNorm	1 × 196608 × 3072
89	Attention_15	Grouped Query Attn	1 × 196608 × 3072
90	Add_15_attn	Add	1 × 196608 × 3072
91	RMSNorm_15_2	RMSNorm	1 × 196608 × 3072
92	MoE_15	MoE Layer	1 × 196608 × 3072
93	Add_15_ffn	Add	1 × 196608 × 3072
94	RMSNorm_16_1	RMSNorm	1 × 196608 × 3072
95	Attention_16	Grouped Query Attn	1 × 196608 × 3072
96	Add_16_attn	Add	1 × 196608 × 3072
97	RMSNorm_16_2	RMSNorm	1 × 196608 × 3072
98	MoE_16	MoE Layer	1 × 196608 × 3072
99	Add_16_ffn	Add	1 × 196608 × 3072
100	RMSNorm_17_1	RMSNorm	1 × 196608 × 3072
101	Attention_17	Grouped Query Attn	1 × 196608 × 3072
102	Add_17_attn	Add	1 × 196608 × 3072
103	RMSNorm_17_2	RMSNorm	1 × 196608 × 3072
104	MoE_17	MoE Layer	1 × 196608 × 3072
105	Add_17_ffn	Add	1 × 196608 × 3072
106	RMSNorm_18_1	RMSNorm	1 × 196608 × 3072
107	Attention_18	Grouped Query Attn	1 × 196608 × 3072
108	Add_18_attn	Add	1 × 196608 × 3072
109	RMSNorm_18_2	RMSNorm	1 × 196608 × 3072
110	MoE_18	MoE Layer	1 × 196608 × 3072
111	Add_18_ffn	Add	1 × 196608 × 3072
112	RMSNorm_19_1	RMSNorm	1 × 196608 × 3072
113	Attention_19	Grouped Query Attn	1 × 196608 × 3072
114	Add_19_attn	Add	1 × 196608 × 3072
115	RMSNorm_19_2	RMSNorm	1 × 196608 × 3072
116	MoE_19	MoE Layer	1 × 196608 × 3072
117	Add_19_ffn	Add	1 × 196608 × 3072
118	RMSNorm_20_1	RMSNorm	1 × 196608 × 3072
119	Attention_20	Grouped Query Attn	1 × 196608 × 3072
120	Add_20_attn	Add	1 × 196608 × 3072
121	RMSNorm_20_2	RMSNorm	1 × 196608 × 3072
122	MoE_20	MoE Layer	1 × 196608 × 3072
123	Add_20_ffn	Add	1 × 196608 × 3072
124	RMSNorm_21_1	RMSNorm	1 × 196608 × 3072
125	Attention_21	Grouped Query Attn	1 × 196608 × 3072
126	Add_21_attn	Add	1 × 196608 × 3072
127	RMSNorm_21_2	RMSNorm	1 × 196608 × 3072
128	MoE_21	MoE Layer	1 × 196608 × 3072
129	Add_21_ffn	Add	1 × 196608 × 3072
130	RMSNorm_22_1	RMSNorm	1 × 196608 × 3072
131	Attention_22	Grouped Query Attn	1 × 196608 × 3072
132	Add_22_attn	Add	1 × 196608 × 3072
133	RMSNorm_22_2	RMSNorm	1 × 196608 × 3072
134	MoE_22	MoE Layer	1 × 196608 × 3072
135	Add_22_ffn	Add	1 × 196608 × 3072
136	RMSNorm_23_1	RMSNorm	1 × 196608 × 3072
137	Attention_23	Grouped Query Attn	1 × 196608 × 3072
138	Add_23_attn	Add	1 × 196608 × 3072
139	RMSNorm_23_2	RMSNorm	1 × 196608 × 3072
140	MoE_23	MoE Layer	1 × 196608 × 3072
141	Add_23_ffn	Add	1 × 196608 × 3072
142	RMSNorm_24_1	RMSNorm	1 × 196608 × 3072
143	Attention_24	Grouped Query Attn	1 × 196608 × 3072
144	Add_24_attn	Add	1 × 196608 × 3072
145	RMSNorm_24_2	RMSNorm	1 × 196608 × 3072
146	MoE_24	MoE Layer	1 × 196608 × 3072
147	Add_24_ffn	Add	1 × 196608 × 3072
148	RMSNorm_25_1	RMSNorm	1 × 196608 × 3072
149	Attention_25	Grouped Query Attn	1 × 196608 × 3072
150	Add_25_attn	Add	1 × 196608 × 3072
151	RMSNorm_25_2	RMSNorm	1 × 196608 × 3072
152	MoE_25	MoE Layer	1 × 196608 × 3072
153	Add_25_ffn	Add	1 × 196608 × 3072
154	RMSNorm_26_1	RMSNorm	1 × 196608 × 3072
155	Attention_26	Grouped Query Attn	1 × 196608 × 3072
156	Add_26_attn	Add	1 × 196608 × 3072
157	RMSNorm_26_2	RMSNorm	1 × 196608 × 3072
158	MoE_26	MoE Layer	1 × 196608 × 3072
159	Add_26_ffn	Add	1 × 196608 × 3072
160	RMSNorm_27_1	RMSNorm	1 × 196608 × 3072
161	Attention_27	Grouped Query Attn	1 × 196608 × 3072
162	Add_27_attn	Add	1 × 196608 × 3072
163	RMSNorm_27_2	RMSNorm	1 × 196608 × 3072
164	MoE_27	MoE Layer	1 × 196608 × 3072
165	Add_27_ffn	Add	1 × 196608 × 3072
166	RMSNorm_28_1	RMSNorm	1 × 196608 × 3072
167	Attention_28	Grouped Query Attn	1 × 196608 × 3072
168	Add_28_attn	Add	1 × 196608 × 3072
169	RMSNorm_28_2	RMSNorm	1 × 196608 × 3072
170	MoE_28	MoE Layer	1 × 196608 × 3072
171	Add_28_ffn	Add	1 × 196608 × 3072
172	RMSNorm_29_1	RMSNorm	1 × 196608 × 3072
173	Attention_29	Grouped Query Attn	1 × 196608 × 3072
174	Add_29_attn	Add	1 × 196608 × 3072
175	RMSNorm_29_2	RMSNorm	1 × 196608 × 3072
176	MoE_29	MoE Layer	1 × 196608 × 3072
177	Add_29_ffn	Add	1 × 196608 × 3072
178	RMSNorm_30_1	RMSNorm	1 × 196608 × 3072
179	Attention_30	Grouped Query Attn	1 × 196608 × 3072
180	Add_30_attn	Add	1 × 196608 × 3072
181	RMSNorm_30_2	RMSNorm	1 × 196608 × 3072
182	MoE_30	MoE Layer	1 × 196608 × 3072
183	Add_30_ffn	Add	1 × 196608 × 3072
184	RMSNorm_31_1	RMSNorm	1 × 196608 × 3072
185	Attention_31	Grouped Query Attn	1 × 196608 × 3072
186	Add_31_attn	Add	1 × 196608 × 3072
187	RMSNorm_31_2	RMSNorm	1 × 196608 × 3072
188	MoE_31	MoE Layer	1 × 196608 × 3072
189	Add_31_ffn	Add	1 × 196608 × 3072
190	RMSNorm_32_1	RMSNorm	1 × 196608 × 3072
191	Attention_32	Grouped Query Attn	1 × 196608 × 3072
192	Add_32_attn	Add	1 × 196608 × 3072
193	RMSNorm_32_2	RMSNorm	1 × 196608 × 3072
194	MoE_32	MoE Layer	1 × 196608 × 3072
195	Add_32_ffn	Add	1 × 196608 × 3072
196	RMSNorm_33_1	RMSNorm	1 × 196608 × 3072
197	Attention_33	Grouped Query Attn	1 × 196608 × 3072
198	Add_33_attn	Add	1 × 196608 × 3072
199	RMSNorm_33_2	RMSNorm	1 × 196608 × 3072
200	MoE_33	MoE Layer	1 × 196608 × 3072
201	Add_33_ffn	Add	1 × 196608 × 3072
202	RMSNorm_34_1	RMSNorm	1 × 196608 × 3072
203	Attention_34	Grouped Query Attn	1 × 196608 × 3072
204	Add_34_attn	Add	1 × 196608 × 3072
205	RMSNorm_34_2	RMSNorm	1 × 196608 × 3072
206	MoE_34	MoE Layer	1 × 196608 × 3072
207	Add_34_ffn	Add	1 × 196608 × 3072
208	RMSNorm_35_1	RMSNorm	1 × 196608 × 3072
209	Attention_35	Grouped Query Attn	1 × 196608 × 3072
210	Add_35_attn	Add	1 × 196608 × 3072
211	RMSNorm_35_2	RMSNorm	1 × 196608 × 3072
212	MoE_35	MoE Layer	1 × 196608 × 3072
213	Add_35_ffn	Add	1 × 196608 × 3072
214	RMSNorm_36_1	RMSNorm	1 × 196608 × 3072
215	Attention_36	Grouped Query Attn	1 × 196608 × 3072
216	Add_36_attn	Add	1 × 196608 × 3072
217	RMSNorm_36_2	RMSNorm	1 × 196608 × 3072
218	MoE_36	MoE Layer	1 × 196608 × 3072
219	Add_36_ffn	Add	1 × 196608 × 3072
220	RMSNorm_37_1	RMSNorm	1 × 196608 × 3072
221	Attention_37	Grouped Query Attn	1 × 196608 × 3072
222	Add_37_attn	Add	1 × 196608 × 3072
223	RMSNorm_37_2	RMSNorm	1 × 196608 × 3072
224	MoE_37	MoE Layer	1 × 196608 × 3072
225	Add_37_ffn	Add	1 × 196608 × 3072
226	RMSNorm_38_1	RMSNorm	1 × 196608 × 3072
227	Attention_38	Grouped Query Attn	1 × 196608 × 3072
228	Add_38_attn	Add	1 × 196608 × 3072
229	RMSNorm_38_2	RMSNorm	1 × 196608 × 3072
230	MoE_38	MoE Layer	1 × 196608 × 3072
231	Add_38_ffn	Add	1 × 196608 × 3072
232	RMSNorm_39_1	RMSNorm	1 × 196608 × 3072
233	Attention_39	Grouped Query Attn	1 × 196608 × 3072
234	Add_39_attn	Add	1 × 196608 × 3072
235	RMSNorm_39_2	RMSNorm	1 × 196608 × 3072
236	MoE_39	MoE Layer	1 × 196608 × 3072
237	Add_39_ffn	Add	1 × 196608 × 3072
238	RMSNorm_40_1	RMSNorm	1 × 196608 × 3072
239	Attention_40	Grouped Query Attn	1 × 196608 × 3072
240	Add_40_attn	Add	1 × 196608 × 3072
241	RMSNorm_40_2	RMSNorm	1 × 196608 × 3072
242	MoE_40	MoE Layer	1 × 196608 × 3072
243	Add_40_ffn	Add	1 × 196608 × 3072
244	RMSNorm_41_1	RMSNorm	1 × 196608 × 3072
245	Attention_41	Grouped Query Attn	1 × 196608 × 3072
246	Add_41_attn	Add	1 × 196608 × 3072
247	RMSNorm_41_2	RMSNorm	1 × 196608 × 3072
248	MoE_41	MoE Layer	1 × 196608 × 3072
249	Add_41_ffn	Add	1 × 196608 × 3072
250	RMSNorm_42_1	RMSNorm	1 × 196608 × 3072
251	Attention_42	Grouped Query Attn	1 × 196608 × 3072
252	Add_42_attn	Add	1 × 196608 × 3072
253	RMSNorm_42_2	RMSNorm	1 × 196608 × 3072
254	MoE_42	MoE Layer	1 × 196608 × 3072
255	Add_42_ffn	Add	1 × 196608 × 3072
256	RMSNorm_43_1	RMSNorm	1 × 196608 × 3072
257	Attention_43	Grouped Query Attn	1 × 196608 × 3072
258	Add_43_attn	Add	1 × 196608 × 3072
259	RMSNorm_43_2	RMSNorm	1 × 196608 × 3072
260	MoE_43	MoE Layer	1 × 196608 × 3072
261	Add_43_ffn	Add	1 × 196608 × 3072
262	RMSNorm_44_1	RMSNorm	1 × 196608 × 3072
263	Attention_44	Grouped Query Attn	1 × 196608 × 3072
264	Add_44_attn	Add	1 × 196608 × 3072
265	RMSNorm_44_2	RMSNorm	1 × 196608 × 3072
266	MoE_44	MoE Layer	1 × 196608 × 3072
267	Add_44_ffn	Add	1 × 196608 × 3072
268	RMSNorm_45_1	RMSNorm	1 × 196608 × 3072
269	Attention_45	Grouped Query Attn	1 × 196608 × 3072
270	Add_45_attn	Add	1 × 196608 × 3072
271	RMSNorm_45_2	RMSNorm	1 × 196608 × 3072
272	MoE_45	MoE Layer	1 × 196608 × 3072
273	Add_45_ffn	Add	1 × 196608 × 3072
274	RMSNorm_46_1	RMSNorm	1 × 196608 × 3072
275	Attention_46	Grouped Query Attn	1 × 196608 × 3072
276	Add_46_attn	Add	1 × 196608 × 3072
277	RMSNorm_46_2	RMSNorm	1 × 196608 × 3072
278	MoE_46	MoE Layer	1 × 196608 × 3072
279	Add_46_ffn	Add	1 × 196608 × 3072
280	RMSNorm_47_1	RMSNorm	1 × 196608 × 3072
281	Attention_47	Grouped Query Attn	1 × 196608 × 3072
282	Add_47_attn	Add	1 × 196608 × 3072
283	RMSNorm_47_2	RMSNorm	1 × 196608 × 3072
284	MoE_47	MoE Layer	1 × 196608 × 3072
285	Add_47_ffn	Add	1 × 196608 × 3072
286	RMSNorm_48_1	RMSNorm	1 × 196608 × 3072
287	Attention_48	Grouped Query Attn	1 × 196608 × 3072
288	Add_48_attn	Add	1 × 196608 × 3072
289	RMSNorm_48_2	RMSNorm	1 × 196608 × 3072
290	MoE_48	MoE Layer	1 × 196608 × 3072
291	Add_48_ffn	Add	1 × 196608 × 3072
292	RMSNorm_49_1	RMSNorm	1 × 196608 × 3072
293	Attention_49	Grouped Query Attn	1 × 196608 × 3072
294	Add_49_attn	Add	1 × 196608 × 3072
295	RMSNorm_49_2	RMSNorm	1 × 196608 × 3072
296	MoE_49	MoE Layer	1 × 196608 × 3072
297	Add_49_ffn	Add	1 × 196608 × 3072
298	RMSNorm_50_1	RMSNorm	1 × 196608 × 3072
299	Attention_50	Grouped Query Attn	1 × 196608 × 3072
300	Add_50_attn	Add	1 × 196608 × 3072
301	RMSNorm_50_2	RMSNorm	1 × 196608 × 3072
302	MoE_50	MoE Layer	1 × 196608 × 3072
303	Add_50_ffn	Add	1 × 196608 × 3072
304	RMSNorm_51_1	RMSNorm	1 × 196608 × 3072
305	Attention_51	Grouped Query Attn	1 × 196608 × 3072
306	Add_51_attn	Add	1 × 196608 × 3072
307	RMSNorm_51_2	RMSNorm	1 × 196608 × 3072
308	MoE_51	MoE Layer	1 × 196608 × 3072
309	Add_51_ffn	Add	1 × 196608 × 3072
310	RMSNorm_52_1	RMSNorm	1 × 196608 × 3072
311	Attention_52	Grouped Query Attn	1 × 196608 × 3072
312	Add_52_attn	Add	1 × 196608 × 3072
313	RMSNorm_52_2	RMSNorm	1 × 196608 × 3072
314	MoE_52	MoE Layer	1 × 196608 × 3072
315	Add_52_ffn	Add	1 × 196608 × 3072
316	RMSNorm_53_1	RMSNorm	1 × 196608 × 3072
317	Attention_53	Grouped Query Attn	1 × 196608 × 3072
318	Add_53_attn	Add	1 × 196608 × 3072
319	RMSNorm_53_2	RMSNorm	1 × 196608 × 3072
320	MoE_53	MoE Layer	1 × 196608 × 3072
321	Add_53_ffn	Add	1 × 196608 × 3072
322	RMSNorm_54_1	RMSNorm	1 × 196608 × 3072
323	Attention_54	Grouped Query Attn	1 × 196608 × 3072
324	Add_54_attn	Add	1 × 196608 × 3072
325	RMSNorm_54_2	RMSNorm	1 × 196608 × 3072
326	MoE_54	MoE Layer	1 × 196608 × 3072
327	Add_54_ffn	Add	1 × 196608 × 3072
328	RMSNorm_55_1	RMSNorm	1 × 196608 × 3072
329	Attention_55	Grouped Query Attn	1 × 196608 × 3072
330	Add_55_attn	Add	1 × 196608 × 3072
331	RMSNorm_55_2	RMSNorm	1 × 196608 × 3072
332	MoE_55	MoE Layer	1 × 196608 × 3072
333	Add_55_ffn	Add	1 × 196608 × 3072
334	RMSNorm_56_1	RMSNorm	1 × 196608 × 3072
335	Attention_56	Grouped Query Attn	1 × 196608 × 3072
336	Add_56_attn	Add	1 × 196608 × 3072
337	RMSNorm_56_2	RMSNorm	1 × 196608 × 3072
338	MoE_56	MoE Layer	1 × 196608 × 3072
339	Add_56_ffn	Add	1 × 196608 × 3072
340	RMSNorm_57_1	RMSNorm	1 × 196608 × 3072
341	Attention_57	Grouped Query Attn	1 × 196608 × 3072
342	Add_57_attn	Add	1 × 196608 × 3072
343	RMSNorm_57_2	RMSNorm	1 × 196608 × 3072
344	MoE_57	MoE Layer	1 × 196608 × 3072
345	Add_57_ffn	Add	1 × 196608 × 3072
346	RMSNorm_58_1	RMSNorm	1 × 196608 × 3072
347	Attention_58	Grouped Query Attn	1 × 196608 × 3072
348	Add_58_attn	Add	1 × 196608 × 3072
349	RMSNorm_58_2	RMSNorm	1 × 196608 × 3072
350	MoE_58	MoE Layer	1 × 196608 × 3072
351	Add_58_ffn	Add	1 × 196608 × 3072
352	RMSNorm_59_1	RMSNorm	1 × 196608 × 3072
353	Attention_59	Grouped Query Attn	1 × 196608 × 3072
354	Add_59_attn	Add	1 × 196608 × 3072
355	RMSNorm_59_2	RMSNorm	1 × 196608 × 3072
356	MoE_59	MoE Layer	1 × 196608 × 3072
357	Add_59_ffn	Add	1 × 196608 × 3072
358	RMSNorm_60_1	RMSNorm	1 × 196608 × 3072
359	Attention_60	Grouped Query Attn	1 × 196608 × 3072
360	Add_60_attn	Add	1 × 196608 × 3072
361	RMSNorm_60_2	RMSNorm	1 × 196608 × 3072
362	MoE_60	MoE Layer	1 × 196608 × 3072
363	Add_60_ffn	Add	1 × 196608 × 3072
364	RMSNorm_61_1	RMSNorm	1 × 196608 × 3072
365	Attention_61	Grouped Query Attn	1 × 196608 × 3072
366	Add_61_attn	Add	1 × 196608 × 3072
367	RMSNorm_61_2	RMSNorm	1 × 196608 × 3072
368	MoE_61	MoE Layer	1 × 196608 × 3072
369	Add_61_ffn	Add	1 × 196608 × 3072
370	RMSNorm_62_1	RMSNorm	1 × 196608 × 3072
371	Attention_62	Grouped Query Attn	1 × 196608 × 3072
372	Add_62_attn	Add	1 × 196608 × 3072
373	RMSNorm_62_2	RMSNorm	1 × 196608 × 3072
374	MoE_62	MoE Layer	1 × 196608 × 3072
375	Add_62_ffn	Add	1 × 196608 × 3072
376	Final_RMSNorm	RMSNorm	1 × 196608 × 3072
377	LM_Head	Linear	1 × 196608 × 200064
378	Output	Output	1 × 196608 × 200064

What the verifier says

infoMoE layers require an auxiliary router z-loss + load-balance loss during training to prevent expert collapse. This is not visible in the architecture diagram but must be in the training loop. Applies to all 62: MoE_1, MoE_2, MoE_3, MoE_4, MoE_5, MoE_6, MoE_7, MoE_8, MoE_9, MoE_10, MoE_11, MoE_12, MoE_13, MoE_14, MoE_15, MoE_16, MoE_17, MoE_18, MoE_19, MoE_20, MoE_21, MoE_22, MoE_23, MoE_24, MoE_25, MoE_26, MoE_27, MoE_28, MoE_29, MoE_30, MoE_31, MoE_32, MoE_33, MoE_34, MoE_35, MoE_36, MoE_37, MoE_38, MoE_39, MoE_40, MoE_41, MoE_42, MoE_43, MoE_44, MoE_45, MoE_46, MoE_47, MoE_48, MoE_49, MoE_50, MoE_51, MoE_52, MoE_53, MoE_54, MoE_55, MoE_56, MoE_57, MoE_58, MoE_59, MoE_60, MoE_61, MoE_62. Fix: Add a note on these layers. Typical aux_loss coefficient: 1e-2 (Mixtral/Switch Transformer).
moe-no-aux-loss

infoAt 62 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace MiniMaxAI/MiniMax-M2 --plan --share

Paste a model instead Markdown twin Index JSON POST /api/v1/plan

Other minimax_m2 checkpoints

MiniMax-M2.5

229B derived · -0.01% against the checkpoint

MiniMax-M2.5-NVFP4

229B derived · +96.6% against the checkpoint

MiniMax-M2.7

229B derived · -0.00% against the checkpoint