Models / internvl_chat

InternVL2-26B

Reconstructed from its own config.json with no weights read. 177K downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure

25.42B

25,417,368,585 parameters

In the published checkpoint

25.51B

25,514,186,112 scalars · safetensors.total, read 2025-03-25

Delta

-0.38%

custom-code This repository ships its own modeling code (`auto_map`, e.g. `configuration_internvl_chat.py`), so `config.json` names a class in the repo rather than an architecture `transformers` defines. The graph below is what those config keys mean under `transformers` semantics, which is not necessarily what the repo's own file builds. A gap here is a statement about what we read, not about the checkpoint.

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32 over 50,000 samples (assumed; no dataset attached). GPU fit is fp32 weights plus gradients plus two Adam moments (16 bytes per parameter) with 1.3x headroom; activations are not included and grow with batch size.

Layers

477

Will it forward-pass

Yes

Priced on

A10G (24GB)

Est. one run

$37223.31

Card	Memory
T4 (16GB)	weights + activations	does not fit
A100 (40GB)	weights + activations	does not fit
H100 (80GB)	weights + activations	does not fit

Structure

480 nodes. Output shapes are propagated from the input shape, batch dimension excluded.

	Layer	Type	Output shape
1	Input	Input	1 × 32768
2	Embedding	Embedding	1 × 32768 × 6144
3	RoPE	RoPE	1 × 32768 × 6144
4	Vision input	Input	3 × 448 × 448
5	PatchEmbed	Patch Embed	1024 × 3200
6	Patch_Position_Embedding	Learned Pos Embed	1024 × 3200
7	Vision_LN_1	LayerNorm	1024 × 3200
8	Vision_Attn_1	Multi-Head Attention	1024 × 3200
9	Vision_Add_1	Add	1024 × 3200
10	Vision_FFN_1	Feed Forward	1024 × 3200
11	Vision_LN_2	LayerNorm	1024 × 3200
12	Vision_Attn_2	Multi-Head Attention	1024 × 3200
13	Vision_Add_2	Add	1024 × 3200
14	Vision_FFN_2	Feed Forward	1024 × 3200
15	Vision_LN_3	LayerNorm	1024 × 3200
16	Vision_Attn_3	Multi-Head Attention	1024 × 3200
17	Vision_Add_3	Add	1024 × 3200
18	Vision_FFN_3	Feed Forward	1024 × 3200
19	Vision_LN_4	LayerNorm	1024 × 3200
20	Vision_Attn_4	Multi-Head Attention	1024 × 3200
21	Vision_Add_4	Add	1024 × 3200
22	Vision_FFN_4	Feed Forward	1024 × 3200
23	Vision_LN_5	LayerNorm	1024 × 3200
24	Vision_Attn_5	Multi-Head Attention	1024 × 3200
25	Vision_Add_5	Add	1024 × 3200
26	Vision_FFN_5	Feed Forward	1024 × 3200
27	Vision_LN_6	LayerNorm	1024 × 3200
28	Vision_Attn_6	Multi-Head Attention	1024 × 3200
29	Vision_Add_6	Add	1024 × 3200
30	Vision_FFN_6	Feed Forward	1024 × 3200
31	Vision_LN_7	LayerNorm	1024 × 3200
32	Vision_Attn_7	Multi-Head Attention	1024 × 3200
33	Vision_Add_7	Add	1024 × 3200
34	Vision_FFN_7	Feed Forward	1024 × 3200
35	Vision_LN_8	LayerNorm	1024 × 3200
36	Vision_Attn_8	Multi-Head Attention	1024 × 3200
37	Vision_Add_8	Add	1024 × 3200
38	Vision_FFN_8	Feed Forward	1024 × 3200
39	Vision_LN_9	LayerNorm	1024 × 3200
40	Vision_Attn_9	Multi-Head Attention	1024 × 3200
41	Vision_Add_9	Add	1024 × 3200
42	Vision_FFN_9	Feed Forward	1024 × 3200
43	Vision_LN_10	LayerNorm	1024 × 3200
44	Vision_Attn_10	Multi-Head Attention	1024 × 3200
45	Vision_Add_10	Add	1024 × 3200
46	Vision_FFN_10	Feed Forward	1024 × 3200
47	Vision_LN_11	LayerNorm	1024 × 3200
48	Vision_Attn_11	Multi-Head Attention	1024 × 3200
49	Vision_Add_11	Add	1024 × 3200
50	Vision_FFN_11	Feed Forward	1024 × 3200
51	Vision_LN_12	LayerNorm	1024 × 3200
52	Vision_Attn_12	Multi-Head Attention	1024 × 3200
53	Vision_Add_12	Add	1024 × 3200
54	Vision_FFN_12	Feed Forward	1024 × 3200
55	Vision_LN_13	LayerNorm	1024 × 3200
56	Vision_Attn_13	Multi-Head Attention	1024 × 3200
57	Vision_Add_13	Add	1024 × 3200
58	Vision_FFN_13	Feed Forward	1024 × 3200
59	Vision_LN_14	LayerNorm	1024 × 3200
60	Vision_Attn_14	Multi-Head Attention	1024 × 3200
61	Vision_Add_14	Add	1024 × 3200
62	Vision_FFN_14	Feed Forward	1024 × 3200
63	Vision_LN_15	LayerNorm	1024 × 3200
64	Vision_Attn_15	Multi-Head Attention	1024 × 3200
65	Vision_Add_15	Add	1024 × 3200
66	Vision_FFN_15	Feed Forward	1024 × 3200
67	Vision_LN_16	LayerNorm	1024 × 3200
68	Vision_Attn_16	Multi-Head Attention	1024 × 3200
69	Vision_Add_16	Add	1024 × 3200
70	Vision_FFN_16	Feed Forward	1024 × 3200
71	Vision_LN_17	LayerNorm	1024 × 3200
72	Vision_Attn_17	Multi-Head Attention	1024 × 3200
73	Vision_Add_17	Add	1024 × 3200
74	Vision_FFN_17	Feed Forward	1024 × 3200
75	Vision_LN_18	LayerNorm	1024 × 3200
76	Vision_Attn_18	Multi-Head Attention	1024 × 3200
77	Vision_Add_18	Add	1024 × 3200
78	Vision_FFN_18	Feed Forward	1024 × 3200
79	Vision_LN_19	LayerNorm	1024 × 3200
80	Vision_Attn_19	Multi-Head Attention	1024 × 3200
81	Vision_Add_19	Add	1024 × 3200
82	Vision_FFN_19	Feed Forward	1024 × 3200
83	Vision_LN_20	LayerNorm	1024 × 3200
84	Vision_Attn_20	Multi-Head Attention	1024 × 3200
85	Vision_Add_20	Add	1024 × 3200
86	Vision_FFN_20	Feed Forward	1024 × 3200
87	Vision_LN_21	LayerNorm	1024 × 3200
88	Vision_Attn_21	Multi-Head Attention	1024 × 3200
89	Vision_Add_21	Add	1024 × 3200
90	Vision_FFN_21	Feed Forward	1024 × 3200
91	Vision_LN_22	LayerNorm	1024 × 3200
92	Vision_Attn_22	Multi-Head Attention	1024 × 3200
93	Vision_Add_22	Add	1024 × 3200
94	Vision_FFN_22	Feed Forward	1024 × 3200
95	Vision_LN_23	LayerNorm	1024 × 3200
96	Vision_Attn_23	Multi-Head Attention	1024 × 3200
97	Vision_Add_23	Add	1024 × 3200
98	Vision_FFN_23	Feed Forward	1024 × 3200
99	Vision_LN_24	LayerNorm	1024 × 3200
100	Vision_Attn_24	Multi-Head Attention	1024 × 3200
101	Vision_Add_24	Add	1024 × 3200
102	Vision_FFN_24	Feed Forward	1024 × 3200
103	Vision_LN_25	LayerNorm	1024 × 3200
104	Vision_Attn_25	Multi-Head Attention	1024 × 3200
105	Vision_Add_25	Add	1024 × 3200
106	Vision_FFN_25	Feed Forward	1024 × 3200
107	Vision_LN_26	LayerNorm	1024 × 3200
108	Vision_Attn_26	Multi-Head Attention	1024 × 3200
109	Vision_Add_26	Add	1024 × 3200
110	Vision_FFN_26	Feed Forward	1024 × 3200
111	Vision_LN_27	LayerNorm	1024 × 3200
112	Vision_Attn_27	Multi-Head Attention	1024 × 3200
113	Vision_Add_27	Add	1024 × 3200
114	Vision_FFN_27	Feed Forward	1024 × 3200
115	Vision_LN_28	LayerNorm	1024 × 3200
116	Vision_Attn_28	Multi-Head Attention	1024 × 3200
117	Vision_Add_28	Add	1024 × 3200
118	Vision_FFN_28	Feed Forward	1024 × 3200
119	Vision_LN_29	LayerNorm	1024 × 3200
120	Vision_Attn_29	Multi-Head Attention	1024 × 3200
121	Vision_Add_29	Add	1024 × 3200
122	Vision_FFN_29	Feed Forward	1024 × 3200
123	Vision_LN_30	LayerNorm	1024 × 3200
124	Vision_Attn_30	Multi-Head Attention	1024 × 3200
125	Vision_Add_30	Add	1024 × 3200
126	Vision_FFN_30	Feed Forward	1024 × 3200
127	Vision_LN_31	LayerNorm	1024 × 3200
128	Vision_Attn_31	Multi-Head Attention	1024 × 3200
129	Vision_Add_31	Add	1024 × 3200
130	Vision_FFN_31	Feed Forward	1024 × 3200
131	Vision_LN_32	LayerNorm	1024 × 3200
132	Vision_Attn_32	Multi-Head Attention	1024 × 3200
133	Vision_Add_32	Add	1024 × 3200
134	Vision_FFN_32	Feed Forward	1024 × 3200
135	Vision_LN_33	LayerNorm	1024 × 3200
136	Vision_Attn_33	Multi-Head Attention	1024 × 3200
137	Vision_Add_33	Add	1024 × 3200
138	Vision_FFN_33	Feed Forward	1024 × 3200
139	Vision_LN_34	LayerNorm	1024 × 3200
140	Vision_Attn_34	Multi-Head Attention	1024 × 3200
141	Vision_Add_34	Add	1024 × 3200
142	Vision_FFN_34	Feed Forward	1024 × 3200
143	Vision_LN_35	LayerNorm	1024 × 3200
144	Vision_Attn_35	Multi-Head Attention	1024 × 3200
145	Vision_Add_35	Add	1024 × 3200
146	Vision_FFN_35	Feed Forward	1024 × 3200
147	Vision_LN_36	LayerNorm	1024 × 3200
148	Vision_Attn_36	Multi-Head Attention	1024 × 3200
149	Vision_Add_36	Add	1024 × 3200
150	Vision_FFN_36	Feed Forward	1024 × 3200
151	Vision_LN_37	LayerNorm	1024 × 3200
152	Vision_Attn_37	Multi-Head Attention	1024 × 3200
153	Vision_Add_37	Add	1024 × 3200
154	Vision_FFN_37	Feed Forward	1024 × 3200
155	Vision_LN_38	LayerNorm	1024 × 3200
156	Vision_Attn_38	Multi-Head Attention	1024 × 3200
157	Vision_Add_38	Add	1024 × 3200
158	Vision_FFN_38	Feed Forward	1024 × 3200
159	Vision_LN_39	LayerNorm	1024 × 3200
160	Vision_Attn_39	Multi-Head Attention	1024 × 3200
161	Vision_Add_39	Add	1024 × 3200
162	Vision_FFN_39	Feed Forward	1024 × 3200
163	Vision_LN_40	LayerNorm	1024 × 3200
164	Vision_Attn_40	Multi-Head Attention	1024 × 3200
165	Vision_Add_40	Add	1024 × 3200
166	Vision_FFN_40	Feed Forward	1024 × 3200
167	Vision_LN_41	LayerNorm	1024 × 3200
168	Vision_Attn_41	Multi-Head Attention	1024 × 3200
169	Vision_Add_41	Add	1024 × 3200
170	Vision_FFN_41	Feed Forward	1024 × 3200
171	Vision_LN_42	LayerNorm	1024 × 3200
172	Vision_Attn_42	Multi-Head Attention	1024 × 3200
173	Vision_Add_42	Add	1024 × 3200
174	Vision_FFN_42	Feed Forward	1024 × 3200
175	Vision_LN_43	LayerNorm	1024 × 3200
176	Vision_Attn_43	Multi-Head Attention	1024 × 3200
177	Vision_Add_43	Add	1024 × 3200
178	Vision_FFN_43	Feed Forward	1024 × 3200
179	Vision_LN_44	LayerNorm	1024 × 3200
180	Vision_Attn_44	Multi-Head Attention	1024 × 3200
181	Vision_Add_44	Add	1024 × 3200
182	Vision_FFN_44	Feed Forward	1024 × 3200
183	Vision_LN_45	LayerNorm	1024 × 3200
184	Vision_Attn_45	Multi-Head Attention	1024 × 3200
185	Vision_Add_45	Add	1024 × 3200
186	Vision_FFN_45	Feed Forward	1024 × 3200
187	Vision projector	Projection	1024 × 6144
188	Vision tokens	Reshape	1 × 1024 × 6144
189	Multimodal fusion (concat tokens)	Concatenate	1 × 33792 × 6144
190	RMSNorm_1_1	RMSNorm	1 × 33792 × 6144
191	Attention_1	Grouped Query Attn	1 × 33792 × 6144
192	Add_1_attn	Add	1 × 33792 × 6144
193	RMSNorm_1_2	RMSNorm	1 × 33792 × 6144
194	FFN_1	SwiGLU	1 × 33792 × 6144
195	Add_1_ffn	Add	1 × 33792 × 6144
196	RMSNorm_2_1	RMSNorm	1 × 33792 × 6144
197	Attention_2	Grouped Query Attn	1 × 33792 × 6144
198	Add_2_attn	Add	1 × 33792 × 6144
199	RMSNorm_2_2	RMSNorm	1 × 33792 × 6144
200	FFN_2	SwiGLU	1 × 33792 × 6144
201	Add_2_ffn	Add	1 × 33792 × 6144
202	RMSNorm_3_1	RMSNorm	1 × 33792 × 6144
203	Attention_3	Grouped Query Attn	1 × 33792 × 6144
204	Add_3_attn	Add	1 × 33792 × 6144
205	RMSNorm_3_2	RMSNorm	1 × 33792 × 6144
206	FFN_3	SwiGLU	1 × 33792 × 6144
207	Add_3_ffn	Add	1 × 33792 × 6144
208	RMSNorm_4_1	RMSNorm	1 × 33792 × 6144
209	Attention_4	Grouped Query Attn	1 × 33792 × 6144
210	Add_4_attn	Add	1 × 33792 × 6144
211	RMSNorm_4_2	RMSNorm	1 × 33792 × 6144
212	FFN_4	SwiGLU	1 × 33792 × 6144
213	Add_4_ffn	Add	1 × 33792 × 6144
214	RMSNorm_5_1	RMSNorm	1 × 33792 × 6144
215	Attention_5	Grouped Query Attn	1 × 33792 × 6144
216	Add_5_attn	Add	1 × 33792 × 6144
217	RMSNorm_5_2	RMSNorm	1 × 33792 × 6144
218	FFN_5	SwiGLU	1 × 33792 × 6144
219	Add_5_ffn	Add	1 × 33792 × 6144
220	RMSNorm_6_1	RMSNorm	1 × 33792 × 6144
221	Attention_6	Grouped Query Attn	1 × 33792 × 6144
222	Add_6_attn	Add	1 × 33792 × 6144
223	RMSNorm_6_2	RMSNorm	1 × 33792 × 6144
224	FFN_6	SwiGLU	1 × 33792 × 6144
225	Add_6_ffn	Add	1 × 33792 × 6144
226	RMSNorm_7_1	RMSNorm	1 × 33792 × 6144
227	Attention_7	Grouped Query Attn	1 × 33792 × 6144
228	Add_7_attn	Add	1 × 33792 × 6144
229	RMSNorm_7_2	RMSNorm	1 × 33792 × 6144
230	FFN_7	SwiGLU	1 × 33792 × 6144
231	Add_7_ffn	Add	1 × 33792 × 6144
232	RMSNorm_8_1	RMSNorm	1 × 33792 × 6144
233	Attention_8	Grouped Query Attn	1 × 33792 × 6144
234	Add_8_attn	Add	1 × 33792 × 6144
235	RMSNorm_8_2	RMSNorm	1 × 33792 × 6144
236	FFN_8	SwiGLU	1 × 33792 × 6144
237	Add_8_ffn	Add	1 × 33792 × 6144
238	RMSNorm_9_1	RMSNorm	1 × 33792 × 6144
239	Attention_9	Grouped Query Attn	1 × 33792 × 6144
240	Add_9_attn	Add	1 × 33792 × 6144
241	RMSNorm_9_2	RMSNorm	1 × 33792 × 6144
242	FFN_9	SwiGLU	1 × 33792 × 6144
243	Add_9_ffn	Add	1 × 33792 × 6144
244	RMSNorm_10_1	RMSNorm	1 × 33792 × 6144
245	Attention_10	Grouped Query Attn	1 × 33792 × 6144
246	Add_10_attn	Add	1 × 33792 × 6144
247	RMSNorm_10_2	RMSNorm	1 × 33792 × 6144
248	FFN_10	SwiGLU	1 × 33792 × 6144
249	Add_10_ffn	Add	1 × 33792 × 6144
250	RMSNorm_11_1	RMSNorm	1 × 33792 × 6144
251	Attention_11	Grouped Query Attn	1 × 33792 × 6144
252	Add_11_attn	Add	1 × 33792 × 6144
253	RMSNorm_11_2	RMSNorm	1 × 33792 × 6144
254	FFN_11	SwiGLU	1 × 33792 × 6144
255	Add_11_ffn	Add	1 × 33792 × 6144
256	RMSNorm_12_1	RMSNorm	1 × 33792 × 6144
257	Attention_12	Grouped Query Attn	1 × 33792 × 6144
258	Add_12_attn	Add	1 × 33792 × 6144
259	RMSNorm_12_2	RMSNorm	1 × 33792 × 6144
260	FFN_12	SwiGLU	1 × 33792 × 6144
261	Add_12_ffn	Add	1 × 33792 × 6144
262	RMSNorm_13_1	RMSNorm	1 × 33792 × 6144
263	Attention_13	Grouped Query Attn	1 × 33792 × 6144
264	Add_13_attn	Add	1 × 33792 × 6144
265	RMSNorm_13_2	RMSNorm	1 × 33792 × 6144
266	FFN_13	SwiGLU	1 × 33792 × 6144
267	Add_13_ffn	Add	1 × 33792 × 6144
268	RMSNorm_14_1	RMSNorm	1 × 33792 × 6144
269	Attention_14	Grouped Query Attn	1 × 33792 × 6144
270	Add_14_attn	Add	1 × 33792 × 6144
271	RMSNorm_14_2	RMSNorm	1 × 33792 × 6144
272	FFN_14	SwiGLU	1 × 33792 × 6144
273	Add_14_ffn	Add	1 × 33792 × 6144
274	RMSNorm_15_1	RMSNorm	1 × 33792 × 6144
275	Attention_15	Grouped Query Attn	1 × 33792 × 6144
276	Add_15_attn	Add	1 × 33792 × 6144
277	RMSNorm_15_2	RMSNorm	1 × 33792 × 6144
278	FFN_15	SwiGLU	1 × 33792 × 6144
279	Add_15_ffn	Add	1 × 33792 × 6144
280	RMSNorm_16_1	RMSNorm	1 × 33792 × 6144
281	Attention_16	Grouped Query Attn	1 × 33792 × 6144
282	Add_16_attn	Add	1 × 33792 × 6144
283	RMSNorm_16_2	RMSNorm	1 × 33792 × 6144
284	FFN_16	SwiGLU	1 × 33792 × 6144
285	Add_16_ffn	Add	1 × 33792 × 6144
286	RMSNorm_17_1	RMSNorm	1 × 33792 × 6144
287	Attention_17	Grouped Query Attn	1 × 33792 × 6144
288	Add_17_attn	Add	1 × 33792 × 6144
289	RMSNorm_17_2	RMSNorm	1 × 33792 × 6144
290	FFN_17	SwiGLU	1 × 33792 × 6144
291	Add_17_ffn	Add	1 × 33792 × 6144
292	RMSNorm_18_1	RMSNorm	1 × 33792 × 6144
293	Attention_18	Grouped Query Attn	1 × 33792 × 6144
294	Add_18_attn	Add	1 × 33792 × 6144
295	RMSNorm_18_2	RMSNorm	1 × 33792 × 6144
296	FFN_18	SwiGLU	1 × 33792 × 6144
297	Add_18_ffn	Add	1 × 33792 × 6144
298	RMSNorm_19_1	RMSNorm	1 × 33792 × 6144
299	Attention_19	Grouped Query Attn	1 × 33792 × 6144
300	Add_19_attn	Add	1 × 33792 × 6144
301	RMSNorm_19_2	RMSNorm	1 × 33792 × 6144
302	FFN_19	SwiGLU	1 × 33792 × 6144
303	Add_19_ffn	Add	1 × 33792 × 6144
304	RMSNorm_20_1	RMSNorm	1 × 33792 × 6144
305	Attention_20	Grouped Query Attn	1 × 33792 × 6144
306	Add_20_attn	Add	1 × 33792 × 6144
307	RMSNorm_20_2	RMSNorm	1 × 33792 × 6144
308	FFN_20	SwiGLU	1 × 33792 × 6144
309	Add_20_ffn	Add	1 × 33792 × 6144
310	RMSNorm_21_1	RMSNorm	1 × 33792 × 6144
311	Attention_21	Grouped Query Attn	1 × 33792 × 6144
312	Add_21_attn	Add	1 × 33792 × 6144
313	RMSNorm_21_2	RMSNorm	1 × 33792 × 6144
314	FFN_21	SwiGLU	1 × 33792 × 6144
315	Add_21_ffn	Add	1 × 33792 × 6144
316	RMSNorm_22_1	RMSNorm	1 × 33792 × 6144
317	Attention_22	Grouped Query Attn	1 × 33792 × 6144
318	Add_22_attn	Add	1 × 33792 × 6144
319	RMSNorm_22_2	RMSNorm	1 × 33792 × 6144
320	FFN_22	SwiGLU	1 × 33792 × 6144
321	Add_22_ffn	Add	1 × 33792 × 6144
322	RMSNorm_23_1	RMSNorm	1 × 33792 × 6144
323	Attention_23	Grouped Query Attn	1 × 33792 × 6144
324	Add_23_attn	Add	1 × 33792 × 6144
325	RMSNorm_23_2	RMSNorm	1 × 33792 × 6144
326	FFN_23	SwiGLU	1 × 33792 × 6144
327	Add_23_ffn	Add	1 × 33792 × 6144
328	RMSNorm_24_1	RMSNorm	1 × 33792 × 6144
329	Attention_24	Grouped Query Attn	1 × 33792 × 6144
330	Add_24_attn	Add	1 × 33792 × 6144
331	RMSNorm_24_2	RMSNorm	1 × 33792 × 6144
332	FFN_24	SwiGLU	1 × 33792 × 6144
333	Add_24_ffn	Add	1 × 33792 × 6144
334	RMSNorm_25_1	RMSNorm	1 × 33792 × 6144
335	Attention_25	Grouped Query Attn	1 × 33792 × 6144
336	Add_25_attn	Add	1 × 33792 × 6144
337	RMSNorm_25_2	RMSNorm	1 × 33792 × 6144
338	FFN_25	SwiGLU	1 × 33792 × 6144
339	Add_25_ffn	Add	1 × 33792 × 6144
340	RMSNorm_26_1	RMSNorm	1 × 33792 × 6144
341	Attention_26	Grouped Query Attn	1 × 33792 × 6144
342	Add_26_attn	Add	1 × 33792 × 6144
343	RMSNorm_26_2	RMSNorm	1 × 33792 × 6144
344	FFN_26	SwiGLU	1 × 33792 × 6144
345	Add_26_ffn	Add	1 × 33792 × 6144
346	RMSNorm_27_1	RMSNorm	1 × 33792 × 6144
347	Attention_27	Grouped Query Attn	1 × 33792 × 6144
348	Add_27_attn	Add	1 × 33792 × 6144
349	RMSNorm_27_2	RMSNorm	1 × 33792 × 6144
350	FFN_27	SwiGLU	1 × 33792 × 6144
351	Add_27_ffn	Add	1 × 33792 × 6144
352	RMSNorm_28_1	RMSNorm	1 × 33792 × 6144
353	Attention_28	Grouped Query Attn	1 × 33792 × 6144
354	Add_28_attn	Add	1 × 33792 × 6144
355	RMSNorm_28_2	RMSNorm	1 × 33792 × 6144
356	FFN_28	SwiGLU	1 × 33792 × 6144
357	Add_28_ffn	Add	1 × 33792 × 6144
358	RMSNorm_29_1	RMSNorm	1 × 33792 × 6144
359	Attention_29	Grouped Query Attn	1 × 33792 × 6144
360	Add_29_attn	Add	1 × 33792 × 6144
361	RMSNorm_29_2	RMSNorm	1 × 33792 × 6144
362	FFN_29	SwiGLU	1 × 33792 × 6144
363	Add_29_ffn	Add	1 × 33792 × 6144
364	RMSNorm_30_1	RMSNorm	1 × 33792 × 6144
365	Attention_30	Grouped Query Attn	1 × 33792 × 6144
366	Add_30_attn	Add	1 × 33792 × 6144
367	RMSNorm_30_2	RMSNorm	1 × 33792 × 6144
368	FFN_30	SwiGLU	1 × 33792 × 6144
369	Add_30_ffn	Add	1 × 33792 × 6144
370	RMSNorm_31_1	RMSNorm	1 × 33792 × 6144
371	Attention_31	Grouped Query Attn	1 × 33792 × 6144
372	Add_31_attn	Add	1 × 33792 × 6144
373	RMSNorm_31_2	RMSNorm	1 × 33792 × 6144
374	FFN_31	SwiGLU	1 × 33792 × 6144
375	Add_31_ffn	Add	1 × 33792 × 6144
376	RMSNorm_32_1	RMSNorm	1 × 33792 × 6144
377	Attention_32	Grouped Query Attn	1 × 33792 × 6144
378	Add_32_attn	Add	1 × 33792 × 6144
379	RMSNorm_32_2	RMSNorm	1 × 33792 × 6144
380	FFN_32	SwiGLU	1 × 33792 × 6144
381	Add_32_ffn	Add	1 × 33792 × 6144
382	RMSNorm_33_1	RMSNorm	1 × 33792 × 6144
383	Attention_33	Grouped Query Attn	1 × 33792 × 6144
384	Add_33_attn	Add	1 × 33792 × 6144
385	RMSNorm_33_2	RMSNorm	1 × 33792 × 6144
386	FFN_33	SwiGLU	1 × 33792 × 6144
387	Add_33_ffn	Add	1 × 33792 × 6144
388	RMSNorm_34_1	RMSNorm	1 × 33792 × 6144
389	Attention_34	Grouped Query Attn	1 × 33792 × 6144
390	Add_34_attn	Add	1 × 33792 × 6144
391	RMSNorm_34_2	RMSNorm	1 × 33792 × 6144
392	FFN_34	SwiGLU	1 × 33792 × 6144
393	Add_34_ffn	Add	1 × 33792 × 6144
394	RMSNorm_35_1	RMSNorm	1 × 33792 × 6144
395	Attention_35	Grouped Query Attn	1 × 33792 × 6144
396	Add_35_attn	Add	1 × 33792 × 6144
397	RMSNorm_35_2	RMSNorm	1 × 33792 × 6144
398	FFN_35	SwiGLU	1 × 33792 × 6144
399	Add_35_ffn	Add	1 × 33792 × 6144
400	RMSNorm_36_1	RMSNorm	1 × 33792 × 6144
401	Attention_36	Grouped Query Attn	1 × 33792 × 6144
402	Add_36_attn	Add	1 × 33792 × 6144
403	RMSNorm_36_2	RMSNorm	1 × 33792 × 6144
404	FFN_36	SwiGLU	1 × 33792 × 6144
405	Add_36_ffn	Add	1 × 33792 × 6144
406	RMSNorm_37_1	RMSNorm	1 × 33792 × 6144
407	Attention_37	Grouped Query Attn	1 × 33792 × 6144
408	Add_37_attn	Add	1 × 33792 × 6144
409	RMSNorm_37_2	RMSNorm	1 × 33792 × 6144
410	FFN_37	SwiGLU	1 × 33792 × 6144
411	Add_37_ffn	Add	1 × 33792 × 6144
412	RMSNorm_38_1	RMSNorm	1 × 33792 × 6144
413	Attention_38	Grouped Query Attn	1 × 33792 × 6144
414	Add_38_attn	Add	1 × 33792 × 6144
415	RMSNorm_38_2	RMSNorm	1 × 33792 × 6144
416	FFN_38	SwiGLU	1 × 33792 × 6144
417	Add_38_ffn	Add	1 × 33792 × 6144
418	RMSNorm_39_1	RMSNorm	1 × 33792 × 6144
419	Attention_39	Grouped Query Attn	1 × 33792 × 6144
420	Add_39_attn	Add	1 × 33792 × 6144
421	RMSNorm_39_2	RMSNorm	1 × 33792 × 6144
422	FFN_39	SwiGLU	1 × 33792 × 6144
423	Add_39_ffn	Add	1 × 33792 × 6144
424	RMSNorm_40_1	RMSNorm	1 × 33792 × 6144
425	Attention_40	Grouped Query Attn	1 × 33792 × 6144
426	Add_40_attn	Add	1 × 33792 × 6144
427	RMSNorm_40_2	RMSNorm	1 × 33792 × 6144
428	FFN_40	SwiGLU	1 × 33792 × 6144
429	Add_40_ffn	Add	1 × 33792 × 6144
430	RMSNorm_41_1	RMSNorm	1 × 33792 × 6144
431	Attention_41	Grouped Query Attn	1 × 33792 × 6144
432	Add_41_attn	Add	1 × 33792 × 6144
433	RMSNorm_41_2	RMSNorm	1 × 33792 × 6144
434	FFN_41	SwiGLU	1 × 33792 × 6144
435	Add_41_ffn	Add	1 × 33792 × 6144
436	RMSNorm_42_1	RMSNorm	1 × 33792 × 6144
437	Attention_42	Grouped Query Attn	1 × 33792 × 6144
438	Add_42_attn	Add	1 × 33792 × 6144
439	RMSNorm_42_2	RMSNorm	1 × 33792 × 6144
440	FFN_42	SwiGLU	1 × 33792 × 6144
441	Add_42_ffn	Add	1 × 33792 × 6144
442	RMSNorm_43_1	RMSNorm	1 × 33792 × 6144
443	Attention_43	Grouped Query Attn	1 × 33792 × 6144
444	Add_43_attn	Add	1 × 33792 × 6144
445	RMSNorm_43_2	RMSNorm	1 × 33792 × 6144
446	FFN_43	SwiGLU	1 × 33792 × 6144
447	Add_43_ffn	Add	1 × 33792 × 6144
448	RMSNorm_44_1	RMSNorm	1 × 33792 × 6144
449	Attention_44	Grouped Query Attn	1 × 33792 × 6144
450	Add_44_attn	Add	1 × 33792 × 6144
451	RMSNorm_44_2	RMSNorm	1 × 33792 × 6144
452	FFN_44	SwiGLU	1 × 33792 × 6144
453	Add_44_ffn	Add	1 × 33792 × 6144
454	RMSNorm_45_1	RMSNorm	1 × 33792 × 6144
455	Attention_45	Grouped Query Attn	1 × 33792 × 6144
456	Add_45_attn	Add	1 × 33792 × 6144
457	RMSNorm_45_2	RMSNorm	1 × 33792 × 6144
458	FFN_45	SwiGLU	1 × 33792 × 6144
459	Add_45_ffn	Add	1 × 33792 × 6144
460	RMSNorm_46_1	RMSNorm	1 × 33792 × 6144
461	Attention_46	Grouped Query Attn	1 × 33792 × 6144
462	Add_46_attn	Add	1 × 33792 × 6144
463	RMSNorm_46_2	RMSNorm	1 × 33792 × 6144
464	FFN_46	SwiGLU	1 × 33792 × 6144
465	Add_46_ffn	Add	1 × 33792 × 6144
466	RMSNorm_47_1	RMSNorm	1 × 33792 × 6144
467	Attention_47	Grouped Query Attn	1 × 33792 × 6144
468	Add_47_attn	Add	1 × 33792 × 6144
469	RMSNorm_47_2	RMSNorm	1 × 33792 × 6144
470	FFN_47	SwiGLU	1 × 33792 × 6144
471	Add_47_ffn	Add	1 × 33792 × 6144
472	RMSNorm_48_1	RMSNorm	1 × 33792 × 6144
473	Attention_48	Grouped Query Attn	1 × 33792 × 6144
474	Add_48_attn	Add	1 × 33792 × 6144
475	RMSNorm_48_2	RMSNorm	1 × 33792 × 6144
476	FFN_48	SwiGLU	1 × 33792 × 6144
477	Add_48_ffn	Add	1 × 33792 × 6144
478	Final_RMSNorm	RMSNorm	1 × 33792 × 6144
479	LM_Head	Linear	1 × 33792 × 92553
480	Output	Output	1 × 33792 × 92553

What the verifier says

warn"RoPE" receives input but its output is not connected. This layer will be unreachable in the forward pass. Fix: Connect the output forward, or add an Output node if this is the final layer.
dead-end

infoAt 93 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init

warnAcross 93 attention layers this design caches 755 KB per token, so a single 8,192-token sequence needs ~6.3 GB of KV cache before weights or activations. That exceeds the 4 GB budget this rule assumes for serving headroom. Fix: Cut KV width: raise the GQA ratio (fewer numKVHeads), switch to MLA, reduce depth or embedDim, or accept a shorter serving context.
kv-cache-context-budget

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace OpenGVLab/InternVL2-26B --plan --share

Paste a model instead Markdown twin Index JSON POST /api/v1/plan

Other internvl_chat checkpoints

InternVL2-1B

935M derived · -0.38% against the checkpoint

InternVL2-2B

2.20B derived · -0.48% against the checkpoint

InternVL2_5-4B

3.70B derived · -0.28% against the checkpoint

InternVL3-1B

935M derived · -0.38% against the checkpoint