N Neurarch Try it See it Changes Bot Pricing API Open app

Models / dinov2

dinov2-small: GPU memory to fine-tune

Fine-tuning dinov2-small at batch 1 takes about 0.69 GB of GPU memory for a full fine-tune (fits one T4 16 GB), and 0.33 GB with LoRA r=16 on a bf16 base (fits one T4 16 GB). Estimated from the model's structure, not measured. What fits on which card.

Reconstructed from its own config.json with no weights read. 4.4M downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure
22.1M
22,056,192 parameters
In the published checkpoint
22.1M
22,056,576 scalars · safetensors.total, read 2026-09-06
Delta
-0.00%

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32 over 50,000 samples (assumed; no dataset attached).

Layers
99
Will it forward-pass
Yes
Priced on
A10G (24GB)
Est. one run
$1.47

Memory: what fits on which card

Largest batch per card. Training is sized at the input shape below per sample.

Full fine-tune, batch 1
0.69 GB
LoRA r=16, batch 1
0.33 GB
CardFull fine-tuneLoRA r=16
T4 16 GBbatch 51batch 52
L4 24 GBbatch 77batch 78
A100 40 GBbatch 130batch 131
A100 / H100 80 GBbatch 262batch 263
H200 141 GBbatch 463batch 465

Training: the input shape in the structure table, AdamW with bf16 autocast, SDPA attention (no score matrix stored), 1.15x headroom on every card. Full fine-tune holds weights, gradients and optimizer state (1.2 x 16 bytes per parameter) plus activations. LoRA is r=16 on every linear layer but the head, base frozen in bf16 (2 bytes per parameter), adapters at full AdamW state, same activations. A two-card split is a contiguous layer cut (pipeline parallel), computed only where one card cannot hold batch 1.

These are estimates from the graph, not measurements. Against measured runs, held out by architecture, the estimate flagged 113 of the 131 jobs that did not fit their card and said "fits" on the other 18. Every measured run.

Structure

101 nodes. Output shapes are propagated from the input shape, batch dimension excluded.

LayerTypeOutput shape
1InputInput3 × 518 × 518
2PatchEmbeddingPatch Embed1370 × 384
3Patch_Position_EmbeddingLearned Pos Embed1370 × 384
4LayerNorm_1_1LayerNorm1370 × 384
5Attention_1Multi-Head Attention1370 × 384
6LayerScale_1_1LayerScale1370 × 384
7Add_1Add1370 × 384
8LayerNorm_1_2LayerNorm1370 × 384
9FFN_1Feed Forward1370 × 384
10LayerScale_1_2LayerScale1370 × 384
11Add_1_2Add1370 × 384
12LayerNorm_2_1LayerNorm1370 × 384
13Attention_2Multi-Head Attention1370 × 384
14LayerScale_2_1LayerScale1370 × 384
15Add_2Add1370 × 384
16LayerNorm_2_2LayerNorm1370 × 384
17FFN_2Feed Forward1370 × 384
18LayerScale_2_2LayerScale1370 × 384
19Add_2_2Add1370 × 384
20LayerNorm_3_1LayerNorm1370 × 384
21Attention_3Multi-Head Attention1370 × 384
22LayerScale_3_1LayerScale1370 × 384
23Add_3Add1370 × 384
24LayerNorm_3_2LayerNorm1370 × 384
25FFN_3Feed Forward1370 × 384
26LayerScale_3_2LayerScale1370 × 384
27Add_3_2Add1370 × 384
28LayerNorm_4_1LayerNorm1370 × 384
29Attention_4Multi-Head Attention1370 × 384
30LayerScale_4_1LayerScale1370 × 384
31Add_4Add1370 × 384
32LayerNorm_4_2LayerNorm1370 × 384
33FFN_4Feed Forward1370 × 384
34LayerScale_4_2LayerScale1370 × 384
35Add_4_2Add1370 × 384
36LayerNorm_5_1LayerNorm1370 × 384
37Attention_5Multi-Head Attention1370 × 384
38LayerScale_5_1LayerScale1370 × 384
39Add_5Add1370 × 384
40LayerNorm_5_2LayerNorm1370 × 384
41FFN_5Feed Forward1370 × 384
42LayerScale_5_2LayerScale1370 × 384
43Add_5_2Add1370 × 384
44LayerNorm_6_1LayerNorm1370 × 384
45Attention_6Multi-Head Attention1370 × 384
46LayerScale_6_1LayerScale1370 × 384
47Add_6Add1370 × 384
48LayerNorm_6_2LayerNorm1370 × 384
49FFN_6Feed Forward1370 × 384
50LayerScale_6_2LayerScale1370 × 384
51Add_6_2Add1370 × 384
52LayerNorm_7_1LayerNorm1370 × 384
53Attention_7Multi-Head Attention1370 × 384
54LayerScale_7_1LayerScale1370 × 384
55Add_7Add1370 × 384
56LayerNorm_7_2LayerNorm1370 × 384
57FFN_7Feed Forward1370 × 384
58LayerScale_7_2LayerScale1370 × 384
59Add_7_2Add1370 × 384
60LayerNorm_8_1LayerNorm1370 × 384
61Attention_8Multi-Head Attention1370 × 384
62LayerScale_8_1LayerScale1370 × 384
63Add_8Add1370 × 384
64LayerNorm_8_2LayerNorm1370 × 384
65FFN_8Feed Forward1370 × 384
66LayerScale_8_2LayerScale1370 × 384
67Add_8_2Add1370 × 384
68LayerNorm_9_1LayerNorm1370 × 384
69Attention_9Multi-Head Attention1370 × 384
70LayerScale_9_1LayerScale1370 × 384
71Add_9Add1370 × 384
72LayerNorm_9_2LayerNorm1370 × 384
73FFN_9Feed Forward1370 × 384
74LayerScale_9_2LayerScale1370 × 384
75Add_9_2Add1370 × 384
76LayerNorm_10_1LayerNorm1370 × 384
77Attention_10Multi-Head Attention1370 × 384
78LayerScale_10_1LayerScale1370 × 384
79Add_10Add1370 × 384
80LayerNorm_10_2LayerNorm1370 × 384
81FFN_10Feed Forward1370 × 384
82LayerScale_10_2LayerScale1370 × 384
83Add_10_2Add1370 × 384
84LayerNorm_11_1LayerNorm1370 × 384
85Attention_11Multi-Head Attention1370 × 384
86LayerScale_11_1LayerScale1370 × 384
87Add_11Add1370 × 384
88LayerNorm_11_2LayerNorm1370 × 384
89FFN_11Feed Forward1370 × 384
90LayerScale_11_2LayerScale1370 × 384
91Add_11_2Add1370 × 384
92LayerNorm_12_1LayerNorm1370 × 384
93Attention_12Multi-Head Attention1370 × 384
94LayerScale_12_1LayerScale1370 × 384
95Add_12Add1370 × 384
96LayerNorm_12_2LayerNorm1370 × 384
97FFN_12Feed Forward1370 × 384
98LayerScale_12_2LayerScale1370 × 384
99Add_12_2Add1370 × 384
100Final_LayerNormLayerNorm1370 × 384
101OutputOutput1370 × 384

What the verifier says

warn"Final_LayerNorm" (layerNorm) is the last layer before Output. Normalizing the raw logits constrains the output range and breaks standard loss functions. Fix: Move normalization before the final Linear/Conv layer.
bn-at-output
infoAt 12 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
deep-attention-default-init

What changed upstream

The change tracker follows facebook/dinov2-small: every revision it has read, what changed in the config, tokenizer, chat template or weights between them, and an OSV advisory per change.

Revision history and changes Badge JSON OSV advisories All tracked changes

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace facebook/dinov2-small --plan --share
Paste a model instead Markdown twin Index JSON POST /api/v1/plan