# CLIP-ViT-L-14-laion2B-s32B-b82K

Hugging Face: `laion/CLIP-ViT-L-14-laion2B-s32B-b82K`
Reconstructed from `config.json` alone. No weights were read.

| | |
|---|---|
| Derived from structure | 302,835,712 |
| In the published checkpoint | 427,616,846 (safetensors.total, 2024-01-16) |
| Delta | -29.2% |
| Comparability | multi-tower |
| Layers | 146 |
| Priced on | A10G (24GB) |
| Est. one run | $3.17 |

> The config declares 2 sub-models (text_config, vision_config). The published checkpoint carries all of them; the graph below carries the towers the reader reconstructs. A gap here is a statement about what we read, not about the checkpoint.

## Structure

| # | Layer | Type | Output shape |
|---|---|---|---|
| 1 | Input | Input | 1 × 3 × 224 × 224 |
| 2 | PatchEmbedding | Embedding | 1 × 3 × 224 × 224 × 1024 |
| 3 | Patch_Position_Embedding | Learned Pos Embed | 1 × 3 × 224 × 224 × 1024 |
| 4 | Attention_1 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 5 | Add_1 | Add | 1 × 3 × 224 × 224 × 1024 |
| 6 | LayerNorm_1_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 7 | FFN_1 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 8 | Add_1_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 9 | LayerNorm_1_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 10 | Attention_2 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 11 | Add_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 12 | LayerNorm_2_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 13 | FFN_2 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 14 | Add_2_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 15 | LayerNorm_2_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 16 | Attention_3 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 17 | Add_3 | Add | 1 × 3 × 224 × 224 × 1024 |
| 18 | LayerNorm_3_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 19 | FFN_3 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 20 | Add_3_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 21 | LayerNorm_3_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 22 | Attention_4 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 23 | Add_4 | Add | 1 × 3 × 224 × 224 × 1024 |
| 24 | LayerNorm_4_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 25 | FFN_4 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 26 | Add_4_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 27 | LayerNorm_4_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 28 | Attention_5 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 29 | Add_5 | Add | 1 × 3 × 224 × 224 × 1024 |
| 30 | LayerNorm_5_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 31 | FFN_5 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 32 | Add_5_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 33 | LayerNorm_5_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 34 | Attention_6 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 35 | Add_6 | Add | 1 × 3 × 224 × 224 × 1024 |
| 36 | LayerNorm_6_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 37 | FFN_6 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 38 | Add_6_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 39 | LayerNorm_6_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 40 | Attention_7 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 41 | Add_7 | Add | 1 × 3 × 224 × 224 × 1024 |
| 42 | LayerNorm_7_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 43 | FFN_7 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 44 | Add_7_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 45 | LayerNorm_7_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 46 | Attention_8 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 47 | Add_8 | Add | 1 × 3 × 224 × 224 × 1024 |
| 48 | LayerNorm_8_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 49 | FFN_8 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 50 | Add_8_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 51 | LayerNorm_8_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 52 | Attention_9 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 53 | Add_9 | Add | 1 × 3 × 224 × 224 × 1024 |
| 54 | LayerNorm_9_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 55 | FFN_9 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 56 | Add_9_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 57 | LayerNorm_9_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 58 | Attention_10 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 59 | Add_10 | Add | 1 × 3 × 224 × 224 × 1024 |
| 60 | LayerNorm_10_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 61 | FFN_10 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 62 | Add_10_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 63 | LayerNorm_10_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 64 | Attention_11 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 65 | Add_11 | Add | 1 × 3 × 224 × 224 × 1024 |
| 66 | LayerNorm_11_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 67 | FFN_11 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 68 | Add_11_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 69 | LayerNorm_11_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 70 | Attention_12 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 71 | Add_12 | Add | 1 × 3 × 224 × 224 × 1024 |
| 72 | LayerNorm_12_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 73 | FFN_12 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 74 | Add_12_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 75 | LayerNorm_12_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 76 | Attention_13 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 77 | Add_13 | Add | 1 × 3 × 224 × 224 × 1024 |
| 78 | LayerNorm_13_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 79 | FFN_13 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 80 | Add_13_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 81 | LayerNorm_13_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 82 | Attention_14 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 83 | Add_14 | Add | 1 × 3 × 224 × 224 × 1024 |
| 84 | LayerNorm_14_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 85 | FFN_14 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 86 | Add_14_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 87 | LayerNorm_14_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 88 | Attention_15 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 89 | Add_15 | Add | 1 × 3 × 224 × 224 × 1024 |
| 90 | LayerNorm_15_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 91 | FFN_15 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 92 | Add_15_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 93 | LayerNorm_15_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 94 | Attention_16 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 95 | Add_16 | Add | 1 × 3 × 224 × 224 × 1024 |
| 96 | LayerNorm_16_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 97 | FFN_16 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 98 | Add_16_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 99 | LayerNorm_16_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 100 | Attention_17 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 101 | Add_17 | Add | 1 × 3 × 224 × 224 × 1024 |
| 102 | LayerNorm_17_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 103 | FFN_17 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 104 | Add_17_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 105 | LayerNorm_17_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 106 | Attention_18 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 107 | Add_18 | Add | 1 × 3 × 224 × 224 × 1024 |
| 108 | LayerNorm_18_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 109 | FFN_18 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 110 | Add_18_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 111 | LayerNorm_18_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 112 | Attention_19 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 113 | Add_19 | Add | 1 × 3 × 224 × 224 × 1024 |
| 114 | LayerNorm_19_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 115 | FFN_19 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 116 | Add_19_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 117 | LayerNorm_19_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 118 | Attention_20 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 119 | Add_20 | Add | 1 × 3 × 224 × 224 × 1024 |
| 120 | LayerNorm_20_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 121 | FFN_20 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 122 | Add_20_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 123 | LayerNorm_20_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 124 | Attention_21 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 125 | Add_21 | Add | 1 × 3 × 224 × 224 × 1024 |
| 126 | LayerNorm_21_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 127 | FFN_21 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 128 | Add_21_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 129 | LayerNorm_21_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 130 | Attention_22 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 131 | Add_22 | Add | 1 × 3 × 224 × 224 × 1024 |
| 132 | LayerNorm_22_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 133 | FFN_22 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 134 | Add_22_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 135 | LayerNorm_22_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 136 | Attention_23 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 137 | Add_23 | Add | 1 × 3 × 224 × 224 × 1024 |
| 138 | LayerNorm_23_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 139 | FFN_23 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 140 | Add_23_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 141 | LayerNorm_23_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 142 | Attention_24 | Multi-Head Attention | 1 × 3 × 224 × 224 × 1024 |
| 143 | Add_24 | Add | 1 × 3 × 224 × 224 × 1024 |
| 144 | LayerNorm_24_1 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 145 | FFN_24 | Feed Forward | 1 × 3 × 224 × 224 × 1024 |
| 146 | Add_24_2 | Add | 1 × 3 × 224 × 224 × 1024 |
| 147 | LayerNorm_24_2 | LayerNorm | 1 × 3 × 224 × 224 × 1024 |
| 148 | Output | Output | 1 × 3 × 224 × 224 × 1024 |

## Do this to your own model

```
pip install neurarch-trace
neurarch-trace laion/CLIP-ViT-L-14-laion2B-s32B-b82K --plan --share
```

- Page: https://neurarch.com/m/clip-vit-l-14-laion2b-s32b-b82k.html
- All models: https://neurarch.com/m/index.json
- The checks: https://neurarch.com/r/index.json