Models / siglip

siglip2-giant-opt-patch16-384: GPU memory to fine-tune

Fine-tuning siglip2-giant-opt-patch16-384 at batch 1 and 512 tokens takes about 37.8 GB of GPU memory for a full fine-tune (fits one A100 / H100 80 GB), and 6.63 GB with LoRA r=16 on a bf16 base (fits one T4 16 GB). Estimated from the model's structure, for the towers the graph reconstructs, not measured. What fits on which card.

Reconstructed from its own config.json with no weights read. 2.5M downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure

1.84B

1,844,534,256 parameters

In the published checkpoint

1.87B

1,871,885,426 scalars · safetensors.total, read 2026-09-06

Delta

-1.46%

multi-tower The config declares 2 sub-models (text_config, vision_config). The published checkpoint carries all of them; the graph below carries the towers the reader reconstructs. A gap here is a statement about what we read, not about the checkpoint.

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32, 512 tokens over 50,000 samples (assumed; no dataset attached).

Layers

329

Will it forward-pass

Yes

Priced on

A10G (24GB)

Est. one run

$43.37

Memory: what fits on which card

Largest batch per card. Training is sized at 512 tokens per sample, serving at 512 tokens per sequence.

Full fine-tune, batch 1 at 512 tokens

37.8 GB

LoRA r=16, batch 1 at 512 tokens

6.63 GB

Serve: bf16 weights

3.69 GB

Serve: KV cache, 512 tokens

0.19 GB

Card	Full fine-tune	LoRA r=16	Serve bf16, 512 tokens
T4 16 GB	does not fit at batch 1	batch 3	batch 53
L4 24 GB	2 cards at batch 1 (111 + 218 layers)	batch 6	batch 90
A100 40 GB	2 cards at batch 1 (273 + 56 layers)	batch 12	batch 164
A100 / H100 80 GB	batch 14	batch 26	batch 347
H200 141 GB	batch 35	batch 48	batch 627

Training: 512 tokens per sample (2048, or the model's context when shorter), AdamW with bf16 autocast, SDPA attention (no score matrix stored), 1.15x headroom on every card. Full fine-tune holds weights, gradients and optimizer state (1.2 x 16 bytes per parameter) plus activations. LoRA is r=16 on every linear layer but the head, base frozen in bf16 (2 bytes per parameter), adapters at full AdamW state, same activations. A two-card split is a contiguous layer cut (pipeline parallel), computed only where one card cannot hold batch 1. Serving: bf16 weights plus the KV cache of 512 tokens per sequence; prefill working set, CUDA context and framework reserve not counted.

These are estimates from the graph, not measurements. Against measured runs, held out by architecture, the estimate flagged 113 of the 131 jobs that did not fit their card and said "fits" on the other 18. Every measured run.

Structure

332 nodes. Output shapes are propagated from the input shape, batch dimension excluded, at 512 tokens: the config's maximum context.

	Layer	Type	Output shape
1	Input	Input	1 × 512
2	Embedding	Embedding	1 × 512 × 1152
3	Positional_Embedding	Learned Pos Embed	1 × 512 × 1152
4	Vision input	Input	3 × 384 × 384
5	PatchEmbed	Patch Embed	576 × 1536
6	Patch_Position_Embedding	Learned Pos Embed	576 × 1536
7	Vision_LN_1	LayerNorm	576 × 1536
8	Vision_Attn_1	Multi-Head Attention	576 × 1536
9	Vision_Add_1	Add	576 × 1536
10	Vision_FFN_1	Feed Forward	576 × 1536
11	Vision_LN_2	LayerNorm	576 × 1536
12	Vision_Attn_2	Multi-Head Attention	576 × 1536
13	Vision_Add_2	Add	576 × 1536
14	Vision_FFN_2	Feed Forward	576 × 1536
15	Vision_LN_3	LayerNorm	576 × 1536
16	Vision_Attn_3	Multi-Head Attention	576 × 1536
17	Vision_Add_3	Add	576 × 1536
18	Vision_FFN_3	Feed Forward	576 × 1536
19	Vision_LN_4	LayerNorm	576 × 1536
20	Vision_Attn_4	Multi-Head Attention	576 × 1536
21	Vision_Add_4	Add	576 × 1536
22	Vision_FFN_4	Feed Forward	576 × 1536
23	Vision_LN_5	LayerNorm	576 × 1536
24	Vision_Attn_5	Multi-Head Attention	576 × 1536
25	Vision_Add_5	Add	576 × 1536
26	Vision_FFN_5	Feed Forward	576 × 1536
27	Vision_LN_6	LayerNorm	576 × 1536
28	Vision_Attn_6	Multi-Head Attention	576 × 1536
29	Vision_Add_6	Add	576 × 1536
30	Vision_FFN_6	Feed Forward	576 × 1536
31	Vision_LN_7	LayerNorm	576 × 1536
32	Vision_Attn_7	Multi-Head Attention	576 × 1536
33	Vision_Add_7	Add	576 × 1536
34	Vision_FFN_7	Feed Forward	576 × 1536
35	Vision_LN_8	LayerNorm	576 × 1536
36	Vision_Attn_8	Multi-Head Attention	576 × 1536
37	Vision_Add_8	Add	576 × 1536
38	Vision_FFN_8	Feed Forward	576 × 1536
39	Vision_LN_9	LayerNorm	576 × 1536
40	Vision_Attn_9	Multi-Head Attention	576 × 1536
41	Vision_Add_9	Add	576 × 1536
42	Vision_FFN_9	Feed Forward	576 × 1536
43	Vision_LN_10	LayerNorm	576 × 1536
44	Vision_Attn_10	Multi-Head Attention	576 × 1536
45	Vision_Add_10	Add	576 × 1536
46	Vision_FFN_10	Feed Forward	576 × 1536
47	Vision_LN_11	LayerNorm	576 × 1536
48	Vision_Attn_11	Multi-Head Attention	576 × 1536
49	Vision_Add_11	Add	576 × 1536
50	Vision_FFN_11	Feed Forward	576 × 1536
51	Vision_LN_12	LayerNorm	576 × 1536
52	Vision_Attn_12	Multi-Head Attention	576 × 1536
53	Vision_Add_12	Add	576 × 1536
54	Vision_FFN_12	Feed Forward	576 × 1536
55	Vision_LN_13	LayerNorm	576 × 1536
56	Vision_Attn_13	Multi-Head Attention	576 × 1536
57	Vision_Add_13	Add	576 × 1536
58	Vision_FFN_13	Feed Forward	576 × 1536
59	Vision_LN_14	LayerNorm	576 × 1536
60	Vision_Attn_14	Multi-Head Attention	576 × 1536
61	Vision_Add_14	Add	576 × 1536
62	Vision_FFN_14	Feed Forward	576 × 1536
63	Vision_LN_15	LayerNorm	576 × 1536
64	Vision_Attn_15	Multi-Head Attention	576 × 1536
65	Vision_Add_15	Add	576 × 1536
66	Vision_FFN_15	Feed Forward	576 × 1536
67	Vision_LN_16	LayerNorm	576 × 1536
68	Vision_Attn_16	Multi-Head Attention	576 × 1536
69	Vision_Add_16	Add	576 × 1536
70	Vision_FFN_16	Feed Forward	576 × 1536
71	Vision_LN_17	LayerNorm	576 × 1536
72	Vision_Attn_17	Multi-Head Attention	576 × 1536
73	Vision_Add_17	Add	576 × 1536
74	Vision_FFN_17	Feed Forward	576 × 1536
75	Vision_LN_18	LayerNorm	576 × 1536
76	Vision_Attn_18	Multi-Head Attention	576 × 1536
77	Vision_Add_18	Add	576 × 1536
78	Vision_FFN_18	Feed Forward	576 × 1536
79	Vision_LN_19	LayerNorm	576 × 1536
80	Vision_Attn_19	Multi-Head Attention	576 × 1536
81	Vision_Add_19	Add	576 × 1536
82	Vision_FFN_19	Feed Forward	576 × 1536
83	Vision_LN_20	LayerNorm	576 × 1536
84	Vision_Attn_20	Multi-Head Attention	576 × 1536
85	Vision_Add_20	Add	576 × 1536
86	Vision_FFN_20	Feed Forward	576 × 1536
87	Vision_LN_21	LayerNorm	576 × 1536
88	Vision_Attn_21	Multi-Head Attention	576 × 1536
89	Vision_Add_21	Add	576 × 1536
90	Vision_FFN_21	Feed Forward	576 × 1536
91	Vision_LN_22	LayerNorm	576 × 1536
92	Vision_Attn_22	Multi-Head Attention	576 × 1536
93	Vision_Add_22	Add	576 × 1536
94	Vision_FFN_22	Feed Forward	576 × 1536
95	Vision_LN_23	LayerNorm	576 × 1536
96	Vision_Attn_23	Multi-Head Attention	576 × 1536
97	Vision_Add_23	Add	576 × 1536
98	Vision_FFN_23	Feed Forward	576 × 1536
99	Vision_LN_24	LayerNorm	576 × 1536
100	Vision_Attn_24	Multi-Head Attention	576 × 1536
101	Vision_Add_24	Add	576 × 1536
102	Vision_FFN_24	Feed Forward	576 × 1536
103	Vision_LN_25	LayerNorm	576 × 1536
104	Vision_Attn_25	Multi-Head Attention	576 × 1536
105	Vision_Add_25	Add	576 × 1536
106	Vision_FFN_25	Feed Forward	576 × 1536
107	Vision_LN_26	LayerNorm	576 × 1536
108	Vision_Attn_26	Multi-Head Attention	576 × 1536
109	Vision_Add_26	Add	576 × 1536
110	Vision_FFN_26	Feed Forward	576 × 1536
111	Vision_LN_27	LayerNorm	576 × 1536
112	Vision_Attn_27	Multi-Head Attention	576 × 1536
113	Vision_Add_27	Add	576 × 1536
114	Vision_FFN_27	Feed Forward	576 × 1536
115	Vision_LN_28	LayerNorm	576 × 1536
116	Vision_Attn_28	Multi-Head Attention	576 × 1536
117	Vision_Add_28	Add	576 × 1536
118	Vision_FFN_28	Feed Forward	576 × 1536
119	Vision_LN_29	LayerNorm	576 × 1536
120	Vision_Attn_29	Multi-Head Attention	576 × 1536
121	Vision_Add_29	Add	576 × 1536
122	Vision_FFN_29	Feed Forward	576 × 1536
123	Vision_LN_30	LayerNorm	576 × 1536
124	Vision_Attn_30	Multi-Head Attention	576 × 1536
125	Vision_Add_30	Add	576 × 1536
126	Vision_FFN_30	Feed Forward	576 × 1536
127	Vision_LN_31	LayerNorm	576 × 1536
128	Vision_Attn_31	Multi-Head Attention	576 × 1536
129	Vision_Add_31	Add	576 × 1536
130	Vision_FFN_31	Feed Forward	576 × 1536
131	Vision_LN_32	LayerNorm	576 × 1536
132	Vision_Attn_32	Multi-Head Attention	576 × 1536
133	Vision_Add_32	Add	576 × 1536
134	Vision_FFN_32	Feed Forward	576 × 1536
135	Vision_LN_33	LayerNorm	576 × 1536
136	Vision_Attn_33	Multi-Head Attention	576 × 1536
137	Vision_Add_33	Add	576 × 1536
138	Vision_FFN_33	Feed Forward	576 × 1536
139	Vision_LN_34	LayerNorm	576 × 1536
140	Vision_Attn_34	Multi-Head Attention	576 × 1536
141	Vision_Add_34	Add	576 × 1536
142	Vision_FFN_34	Feed Forward	576 × 1536
143	Vision_LN_35	LayerNorm	576 × 1536
144	Vision_Attn_35	Multi-Head Attention	576 × 1536
145	Vision_Add_35	Add	576 × 1536
146	Vision_FFN_35	Feed Forward	576 × 1536
147	Vision_LN_36	LayerNorm	576 × 1536
148	Vision_Attn_36	Multi-Head Attention	576 × 1536
149	Vision_Add_36	Add	576 × 1536
150	Vision_FFN_36	Feed Forward	576 × 1536
151	Vision_LN_37	LayerNorm	576 × 1536
152	Vision_Attn_37	Multi-Head Attention	576 × 1536
153	Vision_Add_37	Add	576 × 1536
154	Vision_FFN_37	Feed Forward	576 × 1536
155	Vision_LN_38	LayerNorm	576 × 1536
156	Vision_Attn_38	Multi-Head Attention	576 × 1536
157	Vision_Add_38	Add	576 × 1536
158	Vision_FFN_38	Feed Forward	576 × 1536
159	Vision_LN_39	LayerNorm	576 × 1536
160	Vision_Attn_39	Multi-Head Attention	576 × 1536
161	Vision_Add_39	Add	576 × 1536
162	Vision_FFN_39	Feed Forward	576 × 1536
163	Vision_LN_40	LayerNorm	576 × 1536
164	Vision_Attn_40	Multi-Head Attention	576 × 1536
165	Vision_Add_40	Add	576 × 1536
166	Vision_FFN_40	Feed Forward	576 × 1536
167	Vision projector	Projection	576 × 1152
168	Vision tokens	Reshape	1 × 576 × 1152
169	Multimodal fusion (concat tokens)	Concatenate	1 × 1088 × 1152
170	LayerNorm_1_1	LayerNorm	1 × 1088 × 1152
171	Attention_1	Multi-Head Attention	1 × 1088 × 1152
172	Add_1_attn	Add	1 × 1088 × 1152
173	LayerNorm_1_2	LayerNorm	1 × 1088 × 1152
174	FFN_1	Feed Forward	1 × 1088 × 1152
175	Add_1_ffn	Add	1 × 1088 × 1152
176	LayerNorm_2_1	LayerNorm	1 × 1088 × 1152
177	Attention_2	Multi-Head Attention	1 × 1088 × 1152
178	Add_2_attn	Add	1 × 1088 × 1152
179	LayerNorm_2_2	LayerNorm	1 × 1088 × 1152
180	FFN_2	Feed Forward	1 × 1088 × 1152
181	Add_2_ffn	Add	1 × 1088 × 1152
182	LayerNorm_3_1	LayerNorm	1 × 1088 × 1152
183	Attention_3	Multi-Head Attention	1 × 1088 × 1152
184	Add_3_attn	Add	1 × 1088 × 1152
185	LayerNorm_3_2	LayerNorm	1 × 1088 × 1152
186	FFN_3	Feed Forward	1 × 1088 × 1152
187	Add_3_ffn	Add	1 × 1088 × 1152
188	LayerNorm_4_1	LayerNorm	1 × 1088 × 1152
189	Attention_4	Multi-Head Attention	1 × 1088 × 1152
190	Add_4_attn	Add	1 × 1088 × 1152
191	LayerNorm_4_2	LayerNorm	1 × 1088 × 1152
192	FFN_4	Feed Forward	1 × 1088 × 1152
193	Add_4_ffn	Add	1 × 1088 × 1152
194	LayerNorm_5_1	LayerNorm	1 × 1088 × 1152
195	Attention_5	Multi-Head Attention	1 × 1088 × 1152
196	Add_5_attn	Add	1 × 1088 × 1152
197	LayerNorm_5_2	LayerNorm	1 × 1088 × 1152
198	FFN_5	Feed Forward	1 × 1088 × 1152
199	Add_5_ffn	Add	1 × 1088 × 1152
200	LayerNorm_6_1	LayerNorm	1 × 1088 × 1152
201	Attention_6	Multi-Head Attention	1 × 1088 × 1152
202	Add_6_attn	Add	1 × 1088 × 1152
203	LayerNorm_6_2	LayerNorm	1 × 1088 × 1152
204	FFN_6	Feed Forward	1 × 1088 × 1152
205	Add_6_ffn	Add	1 × 1088 × 1152
206	LayerNorm_7_1	LayerNorm	1 × 1088 × 1152
207	Attention_7	Multi-Head Attention	1 × 1088 × 1152
208	Add_7_attn	Add	1 × 1088 × 1152
209	LayerNorm_7_2	LayerNorm	1 × 1088 × 1152
210	FFN_7	Feed Forward	1 × 1088 × 1152
211	Add_7_ffn	Add	1 × 1088 × 1152
212	LayerNorm_8_1	LayerNorm	1 × 1088 × 1152
213	Attention_8	Multi-Head Attention	1 × 1088 × 1152
214	Add_8_attn	Add	1 × 1088 × 1152
215	LayerNorm_8_2	LayerNorm	1 × 1088 × 1152
216	FFN_8	Feed Forward	1 × 1088 × 1152
217	Add_8_ffn	Add	1 × 1088 × 1152
218	LayerNorm_9_1	LayerNorm	1 × 1088 × 1152
219	Attention_9	Multi-Head Attention	1 × 1088 × 1152
220	Add_9_attn	Add	1 × 1088 × 1152
221	LayerNorm_9_2	LayerNorm	1 × 1088 × 1152
222	FFN_9	Feed Forward	1 × 1088 × 1152
223	Add_9_ffn	Add	1 × 1088 × 1152
224	LayerNorm_10_1	LayerNorm	1 × 1088 × 1152
225	Attention_10	Multi-Head Attention	1 × 1088 × 1152
226	Add_10_attn	Add	1 × 1088 × 1152
227	LayerNorm_10_2	LayerNorm	1 × 1088 × 1152
228	FFN_10	Feed Forward	1 × 1088 × 1152
229	Add_10_ffn	Add	1 × 1088 × 1152
230	LayerNorm_11_1	LayerNorm	1 × 1088 × 1152
231	Attention_11	Multi-Head Attention	1 × 1088 × 1152
232	Add_11_attn	Add	1 × 1088 × 1152
233	LayerNorm_11_2	LayerNorm	1 × 1088 × 1152
234	FFN_11	Feed Forward	1 × 1088 × 1152
235	Add_11_ffn	Add	1 × 1088 × 1152
236	LayerNorm_12_1	LayerNorm	1 × 1088 × 1152
237	Attention_12	Multi-Head Attention	1 × 1088 × 1152
238	Add_12_attn	Add	1 × 1088 × 1152
239	LayerNorm_12_2	LayerNorm	1 × 1088 × 1152
240	FFN_12	Feed Forward	1 × 1088 × 1152
241	Add_12_ffn	Add	1 × 1088 × 1152
242	LayerNorm_13_1	LayerNorm	1 × 1088 × 1152
243	Attention_13	Multi-Head Attention	1 × 1088 × 1152
244	Add_13_attn	Add	1 × 1088 × 1152
245	LayerNorm_13_2	LayerNorm	1 × 1088 × 1152
246	FFN_13	Feed Forward	1 × 1088 × 1152
247	Add_13_ffn	Add	1 × 1088 × 1152
248	LayerNorm_14_1	LayerNorm	1 × 1088 × 1152
249	Attention_14	Multi-Head Attention	1 × 1088 × 1152
250	Add_14_attn	Add	1 × 1088 × 1152
251	LayerNorm_14_2	LayerNorm	1 × 1088 × 1152
252	FFN_14	Feed Forward	1 × 1088 × 1152
253	Add_14_ffn	Add	1 × 1088 × 1152
254	LayerNorm_15_1	LayerNorm	1 × 1088 × 1152
255	Attention_15	Multi-Head Attention	1 × 1088 × 1152
256	Add_15_attn	Add	1 × 1088 × 1152
257	LayerNorm_15_2	LayerNorm	1 × 1088 × 1152
258	FFN_15	Feed Forward	1 × 1088 × 1152
259	Add_15_ffn	Add	1 × 1088 × 1152
260	LayerNorm_16_1	LayerNorm	1 × 1088 × 1152
261	Attention_16	Multi-Head Attention	1 × 1088 × 1152
262	Add_16_attn	Add	1 × 1088 × 1152
263	LayerNorm_16_2	LayerNorm	1 × 1088 × 1152
264	FFN_16	Feed Forward	1 × 1088 × 1152
265	Add_16_ffn	Add	1 × 1088 × 1152
266	LayerNorm_17_1	LayerNorm	1 × 1088 × 1152
267	Attention_17	Multi-Head Attention	1 × 1088 × 1152
268	Add_17_attn	Add	1 × 1088 × 1152
269	LayerNorm_17_2	LayerNorm	1 × 1088 × 1152
270	FFN_17	Feed Forward	1 × 1088 × 1152
271	Add_17_ffn	Add	1 × 1088 × 1152
272	LayerNorm_18_1	LayerNorm	1 × 1088 × 1152
273	Attention_18	Multi-Head Attention	1 × 1088 × 1152
274	Add_18_attn	Add	1 × 1088 × 1152
275	LayerNorm_18_2	LayerNorm	1 × 1088 × 1152
276	FFN_18	Feed Forward	1 × 1088 × 1152
277	Add_18_ffn	Add	1 × 1088 × 1152
278	LayerNorm_19_1	LayerNorm	1 × 1088 × 1152
279	Attention_19	Multi-Head Attention	1 × 1088 × 1152
280	Add_19_attn	Add	1 × 1088 × 1152
281	LayerNorm_19_2	LayerNorm	1 × 1088 × 1152
282	FFN_19	Feed Forward	1 × 1088 × 1152
283	Add_19_ffn	Add	1 × 1088 × 1152
284	LayerNorm_20_1	LayerNorm	1 × 1088 × 1152
285	Attention_20	Multi-Head Attention	1 × 1088 × 1152
286	Add_20_attn	Add	1 × 1088 × 1152
287	LayerNorm_20_2	LayerNorm	1 × 1088 × 1152
288	FFN_20	Feed Forward	1 × 1088 × 1152
289	Add_20_ffn	Add	1 × 1088 × 1152
290	LayerNorm_21_1	LayerNorm	1 × 1088 × 1152
291	Attention_21	Multi-Head Attention	1 × 1088 × 1152
292	Add_21_attn	Add	1 × 1088 × 1152
293	LayerNorm_21_2	LayerNorm	1 × 1088 × 1152
294	FFN_21	Feed Forward	1 × 1088 × 1152
295	Add_21_ffn	Add	1 × 1088 × 1152
296	LayerNorm_22_1	LayerNorm	1 × 1088 × 1152
297	Attention_22	Multi-Head Attention	1 × 1088 × 1152
298	Add_22_attn	Add	1 × 1088 × 1152
299	LayerNorm_22_2	LayerNorm	1 × 1088 × 1152
300	FFN_22	Feed Forward	1 × 1088 × 1152
301	Add_22_ffn	Add	1 × 1088 × 1152
302	LayerNorm_23_1	LayerNorm	1 × 1088 × 1152
303	Attention_23	Multi-Head Attention	1 × 1088 × 1152
304	Add_23_attn	Add	1 × 1088 × 1152
305	LayerNorm_23_2	LayerNorm	1 × 1088 × 1152
306	FFN_23	Feed Forward	1 × 1088 × 1152
307	Add_23_ffn	Add	1 × 1088 × 1152
308	LayerNorm_24_1	LayerNorm	1 × 1088 × 1152
309	Attention_24	Multi-Head Attention	1 × 1088 × 1152
310	Add_24_attn	Add	1 × 1088 × 1152
311	LayerNorm_24_2	LayerNorm	1 × 1088 × 1152
312	FFN_24	Feed Forward	1 × 1088 × 1152
313	Add_24_ffn	Add	1 × 1088 × 1152
314	LayerNorm_25_1	LayerNorm	1 × 1088 × 1152
315	Attention_25	Multi-Head Attention	1 × 1088 × 1152
316	Add_25_attn	Add	1 × 1088 × 1152
317	LayerNorm_25_2	LayerNorm	1 × 1088 × 1152
318	FFN_25	Feed Forward	1 × 1088 × 1152
319	Add_25_ffn	Add	1 × 1088 × 1152
320	LayerNorm_26_1	LayerNorm	1 × 1088 × 1152
321	Attention_26	Multi-Head Attention	1 × 1088 × 1152
322	Add_26_attn	Add	1 × 1088 × 1152
323	LayerNorm_26_2	LayerNorm	1 × 1088 × 1152
324	FFN_26	Feed Forward	1 × 1088 × 1152
325	Add_26_ffn	Add	1 × 1088 × 1152
326	LayerNorm_27_1	LayerNorm	1 × 1088 × 1152
327	Attention_27	Multi-Head Attention	1 × 1088 × 1152
328	Add_27_attn	Add	1 × 1088 × 1152
329	LayerNorm_27_2	LayerNorm	1 × 1088 × 1152
330	FFN_27	Feed Forward	1 × 1088 × 1152
331	Add_27_ffn	Add	1 × 1088 × 1152
332	Output	Output	1 × 1088 × 1152

What the verifier says

warn"Positional_Embedding" receives input but its output is not connected. This layer will be unreachable in the forward pass. Fix: Connect the output forward, or add an Output node if this is the final layer.
dead-end

infoAt 67 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init

What changed upstream

The change tracker follows google/siglip2-giant-opt-patch16-384: every revision it has read, what changed in the config, tokenizer, chat template or weights between them, and an OSV advisory per change.

Revision history and changes Badge JSON OSV advisories All tracked changes

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace google/siglip2-giant-opt-patch16-384 --plan --share

Paste a model instead Markdown twin Index JSON POST /api/v1/plan