N Neurarch Try it See it Changes Bot Pricing API Open app

Models / clip

CLIP-ViT-L-14-laion2B-s32B-b82K: GPU memory to fine-tune

Fine-tuning CLIP-ViT-L-14-laion2B-s32B-b82K at batch 1 takes about 6.06 GB of GPU memory for a full fine-tune (fits one T4 16 GB), and 0.98 GB with LoRA r=16 on a bf16 base (fits one T4 16 GB). Estimated from the model's structure, for the towers the graph reconstructs, not measured. What fits on which card.

Reconstructed from its own config.json with no weights read. 3.7M downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure
303M
303,176,704 parameters
In the published checkpoint
428M
427,616,846 scalars · safetensors.total, read 2024-01-16
Delta
-29.1%

multi-tower The config declares 2 sub-models (text_config, vision_config). The published checkpoint carries all of them; the graph below carries the towers the reader reconstructs. A gap here is a statement about what we read, not about the checkpoint.

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32 over 50,000 samples (assumed; no dataset attached).

Layers
146
Will it forward-pass
Yes
Priced on
A10G (24GB)
Est. one run
$3.01

Memory: what fits on which card

Largest batch per card. Training is sized at the input shape below per sample.

Full fine-tune, batch 1
6.06 GB
LoRA r=16, batch 1
0.98 GB
CardFull fine-tuneLoRA r=16
T4 16 GBbatch 33batch 55
L4 24 GBbatch 63batch 84
A100 40 GBbatch 121batch 142
A100 / H100 80 GBbatch 267batch 289
H200 141 GBbatch 490batch 511

Training: the input shape in the structure table, AdamW with bf16 autocast, SDPA attention (no score matrix stored), 1.15x headroom on every card. Full fine-tune holds weights, gradients and optimizer state (1.2 x 16 bytes per parameter) plus activations. LoRA is r=16 on every linear layer but the head, base frozen in bf16 (2 bytes per parameter), adapters at full AdamW state, same activations. A two-card split is a contiguous layer cut (pipeline parallel), computed only where one card cannot hold batch 1.

These are estimates from the graph, not measurements. Against measured runs, held out by architecture, the estimate flagged 113 of the 131 jobs that did not fit their card and said "fits" on the other 18. Every measured run.

Structure

148 nodes. Output shapes are propagated from the input shape, batch dimension excluded.

LayerTypeOutput shape
1InputInput3 × 224 × 224
2PatchEmbeddingPatch Embed257 × 1024
3Patch_Position_EmbeddingLearned Pos Embed257 × 1024
4Attention_1Multi-Head Attention257 × 1024
5Add_1Add257 × 1024
6LayerNorm_1_1LayerNorm257 × 1024
7FFN_1Feed Forward257 × 1024
8Add_1_2Add257 × 1024
9LayerNorm_1_2LayerNorm257 × 1024
10Attention_2Multi-Head Attention257 × 1024
11Add_2Add257 × 1024
12LayerNorm_2_1LayerNorm257 × 1024
13FFN_2Feed Forward257 × 1024
14Add_2_2Add257 × 1024
15LayerNorm_2_2LayerNorm257 × 1024
16Attention_3Multi-Head Attention257 × 1024
17Add_3Add257 × 1024
18LayerNorm_3_1LayerNorm257 × 1024
19FFN_3Feed Forward257 × 1024
20Add_3_2Add257 × 1024
21LayerNorm_3_2LayerNorm257 × 1024
22Attention_4Multi-Head Attention257 × 1024
23Add_4Add257 × 1024
24LayerNorm_4_1LayerNorm257 × 1024
25FFN_4Feed Forward257 × 1024
26Add_4_2Add257 × 1024
27LayerNorm_4_2LayerNorm257 × 1024
28Attention_5Multi-Head Attention257 × 1024
29Add_5Add257 × 1024
30LayerNorm_5_1LayerNorm257 × 1024
31FFN_5Feed Forward257 × 1024
32Add_5_2Add257 × 1024
33LayerNorm_5_2LayerNorm257 × 1024
34Attention_6Multi-Head Attention257 × 1024
35Add_6Add257 × 1024
36LayerNorm_6_1LayerNorm257 × 1024
37FFN_6Feed Forward257 × 1024
38Add_6_2Add257 × 1024
39LayerNorm_6_2LayerNorm257 × 1024
40Attention_7Multi-Head Attention257 × 1024
41Add_7Add257 × 1024
42LayerNorm_7_1LayerNorm257 × 1024
43FFN_7Feed Forward257 × 1024
44Add_7_2Add257 × 1024
45LayerNorm_7_2LayerNorm257 × 1024
46Attention_8Multi-Head Attention257 × 1024
47Add_8Add257 × 1024
48LayerNorm_8_1LayerNorm257 × 1024
49FFN_8Feed Forward257 × 1024
50Add_8_2Add257 × 1024
51LayerNorm_8_2LayerNorm257 × 1024
52Attention_9Multi-Head Attention257 × 1024
53Add_9Add257 × 1024
54LayerNorm_9_1LayerNorm257 × 1024
55FFN_9Feed Forward257 × 1024
56Add_9_2Add257 × 1024
57LayerNorm_9_2LayerNorm257 × 1024
58Attention_10Multi-Head Attention257 × 1024
59Add_10Add257 × 1024
60LayerNorm_10_1LayerNorm257 × 1024
61FFN_10Feed Forward257 × 1024
62Add_10_2Add257 × 1024
63LayerNorm_10_2LayerNorm257 × 1024
64Attention_11Multi-Head Attention257 × 1024
65Add_11Add257 × 1024
66LayerNorm_11_1LayerNorm257 × 1024
67FFN_11Feed Forward257 × 1024
68Add_11_2Add257 × 1024
69LayerNorm_11_2LayerNorm257 × 1024
70Attention_12Multi-Head Attention257 × 1024
71Add_12Add257 × 1024
72LayerNorm_12_1LayerNorm257 × 1024
73FFN_12Feed Forward257 × 1024
74Add_12_2Add257 × 1024
75LayerNorm_12_2LayerNorm257 × 1024
76Attention_13Multi-Head Attention257 × 1024
77Add_13Add257 × 1024
78LayerNorm_13_1LayerNorm257 × 1024
79FFN_13Feed Forward257 × 1024
80Add_13_2Add257 × 1024
81LayerNorm_13_2LayerNorm257 × 1024
82Attention_14Multi-Head Attention257 × 1024
83Add_14Add257 × 1024
84LayerNorm_14_1LayerNorm257 × 1024
85FFN_14Feed Forward257 × 1024
86Add_14_2Add257 × 1024
87LayerNorm_14_2LayerNorm257 × 1024
88Attention_15Multi-Head Attention257 × 1024
89Add_15Add257 × 1024
90LayerNorm_15_1LayerNorm257 × 1024
91FFN_15Feed Forward257 × 1024
92Add_15_2Add257 × 1024
93LayerNorm_15_2LayerNorm257 × 1024
94Attention_16Multi-Head Attention257 × 1024
95Add_16Add257 × 1024
96LayerNorm_16_1LayerNorm257 × 1024
97FFN_16Feed Forward257 × 1024
98Add_16_2Add257 × 1024
99LayerNorm_16_2LayerNorm257 × 1024
100Attention_17Multi-Head Attention257 × 1024
101Add_17Add257 × 1024
102LayerNorm_17_1LayerNorm257 × 1024
103FFN_17Feed Forward257 × 1024
104Add_17_2Add257 × 1024
105LayerNorm_17_2LayerNorm257 × 1024
106Attention_18Multi-Head Attention257 × 1024
107Add_18Add257 × 1024
108LayerNorm_18_1LayerNorm257 × 1024
109FFN_18Feed Forward257 × 1024
110Add_18_2Add257 × 1024
111LayerNorm_18_2LayerNorm257 × 1024
112Attention_19Multi-Head Attention257 × 1024
113Add_19Add257 × 1024
114LayerNorm_19_1LayerNorm257 × 1024
115FFN_19Feed Forward257 × 1024
116Add_19_2Add257 × 1024
117LayerNorm_19_2LayerNorm257 × 1024
118Attention_20Multi-Head Attention257 × 1024
119Add_20Add257 × 1024
120LayerNorm_20_1LayerNorm257 × 1024
121FFN_20Feed Forward257 × 1024
122Add_20_2Add257 × 1024
123LayerNorm_20_2LayerNorm257 × 1024
124Attention_21Multi-Head Attention257 × 1024
125Add_21Add257 × 1024
126LayerNorm_21_1LayerNorm257 × 1024
127FFN_21Feed Forward257 × 1024
128Add_21_2Add257 × 1024
129LayerNorm_21_2LayerNorm257 × 1024
130Attention_22Multi-Head Attention257 × 1024
131Add_22Add257 × 1024
132LayerNorm_22_1LayerNorm257 × 1024
133FFN_22Feed Forward257 × 1024
134Add_22_2Add257 × 1024
135LayerNorm_22_2LayerNorm257 × 1024
136Attention_23Multi-Head Attention257 × 1024
137Add_23Add257 × 1024
138LayerNorm_23_1LayerNorm257 × 1024
139FFN_23Feed Forward257 × 1024
140Add_23_2Add257 × 1024
141LayerNorm_23_2LayerNorm257 × 1024
142Attention_24Multi-Head Attention257 × 1024
143Add_24Add257 × 1024
144LayerNorm_24_1LayerNorm257 × 1024
145FFN_24Feed Forward257 × 1024
146Add_24_2Add257 × 1024
147LayerNorm_24_2LayerNorm257 × 1024
148OutputOutput257 × 1024

What the verifier says

warn"LayerNorm_24_2" (layerNorm) is the last layer before Output. Normalizing the raw logits constrains the output range and breaks standard loss functions. Fix: Move normalization before the final Linear/Conv layer.
bn-at-output
infoAt 24 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init

What changed upstream

The change tracker follows laion/CLIP-ViT-L-14-laion2B-s32B-b82K: every revision it has read, what changed in the config, tokenizer, chat template or weights between them, and an OSV advisory per change.

Revision history and changes Badge JSON OSV advisories All tracked changes

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace laion/CLIP-ViT-L-14-laion2B-s32B-b82K --plan --share
Paste a model instead Markdown twin Index JSON POST /api/v1/plan

Other clip checkpoints

CLIP-ViT-B-32-laion2B-s34B-b79K
87.5M derived · -42.2% against the checkpoint
clip-vit-large-patch14
303M derived · -29.1% against the checkpoint