# LLaVA-1.5 vs CLIP ViT-B/32

A vision-language model against the encoder it is built on.

**CLIP ViT-B/32 has 6.9B fewer parameters than LLaVA-1.5: 7 layers added, 196 removed, 32 changed.**

Source: https://neurarch.com/diff/llava-1.5-7b-vs-clip-vit-b32.html

## Sides

| | LLaVA-1.5 | CLIP ViT-B/32 |
|---|---|---|
| Layers | 226 | 37 |
| Parameters | 7.1B | 151M |
| Input | 3 × 336 × 336 | 3 × 224 × 224 |
| Output | 1 × 2624 × 32000 | 512 |
| Forward-passes | yes | yes |
| Est. train cost | $690.06 | $0.313 |
| T4 16GB | no | fits |
| A100 40GB | no | fits |
| H100 80GB | no | fits |

## Deltas (CLIP ViT-B/32 relative to LLaVA-1.5)

- Parameters: -6.9B (-97.9%)
- Layers: -189
- Added 7, removed 196, changed 32, unchanged 1

## Layer by layer

| # | Status | LLaVA-1.5 | Params | Output | CLIP ViT-B/32 | Params | Output |
|---|---|---|---|---|---|---|---|
| 1 | changed (shape) | image (Input) |  | 3 × 336 × 336 | image (Input) |  | 3 × 224 × 224 |
| 2 | changed (shape) | text_tokens (Input) |  | 1 × 2048 | tokens (Input) |  | 77 |
| 3 | changed (imgSize, patchSize, embedDim) | clip_patch_14 (Patch Embed) | 603K | 576 × 1024 | patch_embed (Patch Embed) | 2.4M | 49 × 768 |
| 4 | changed (numEmbeddings, embeddingDim) | token_embed (Embedding) | 131M | 1 × 2048 × 4096 | tok_embed (Embedding) | 25M | 77 × 512 |
| 5 | added |  | |  | vis_pos (Positional Encoding) |  | 49 × 768 |
| 6 | changed (maxLen, embedDim) | vis_pos (Positional Encoding) |  | 576 × 1024 | txt_pos (Positional Encoding) |  | 77 × 512 |
| 7 | changed (embedDim, numHeads, ffDim) | clip_block_1 (Transformer Block) | 13M | 576 × 1024 | vis_block_1 (Transformer Block) | 7.1M | 49 × 768 |
| 8 | changed (embedDim, numHeads, ffDim) | clip_block_2 (Transformer Block) | 13M | 576 × 1024 | txt_block_1 (Transformer Block) | 3.2M | 77 × 512 |
| 9 | changed (embedDim, numHeads, ffDim) | clip_block_3 (Transformer Block) | 13M | 576 × 1024 | vis_block_2 (Transformer Block) | 7.1M | 49 × 768 |
| 10 | changed (embedDim, numHeads, ffDim) | clip_block_4 (Transformer Block) | 13M | 576 × 1024 | txt_block_2 (Transformer Block) | 3.2M | 77 × 512 |
| 11 | changed (embedDim, numHeads, ffDim) | clip_block_5 (Transformer Block) | 13M | 576 × 1024 | vis_block_3 (Transformer Block) | 7.1M | 49 × 768 |
| 12 | changed (embedDim, numHeads, ffDim) | clip_block_6 (Transformer Block) | 13M | 576 × 1024 | txt_block_3 (Transformer Block) | 3.2M | 77 × 512 |
| 13 | changed (embedDim, numHeads, ffDim) | clip_block_7 (Transformer Block) | 13M | 576 × 1024 | vis_block_4 (Transformer Block) | 7.1M | 49 × 768 |
| 14 | changed (embedDim, numHeads, ffDim) | clip_block_8 (Transformer Block) | 13M | 576 × 1024 | txt_block_4 (Transformer Block) | 3.2M | 77 × 512 |
| 15 | changed (embedDim, numHeads, ffDim) | clip_block_9 (Transformer Block) | 13M | 576 × 1024 | vis_block_5 (Transformer Block) | 7.1M | 49 × 768 |
| 16 | changed (embedDim, numHeads, ffDim) | clip_block_10 (Transformer Block) | 13M | 576 × 1024 | txt_block_5 (Transformer Block) | 3.2M | 77 × 512 |
| 17 | changed (embedDim, numHeads, ffDim) | clip_block_11 (Transformer Block) | 13M | 576 × 1024 | vis_block_6 (Transformer Block) | 7.1M | 49 × 768 |
| 18 | changed (embedDim, numHeads, ffDim) | clip_block_12 (Transformer Block) | 13M | 576 × 1024 | txt_block_6 (Transformer Block) | 3.2M | 77 × 512 |
| 19 | changed (embedDim, numHeads, ffDim) | clip_block_13 (Transformer Block) | 13M | 576 × 1024 | vis_block_7 (Transformer Block) | 7.1M | 49 × 768 |
| 20 | changed (embedDim, numHeads, ffDim) | clip_block_14 (Transformer Block) | 13M | 576 × 1024 | txt_block_7 (Transformer Block) | 3.2M | 77 × 512 |
| 21 | changed (embedDim, numHeads, ffDim) | clip_block_15 (Transformer Block) | 13M | 576 × 1024 | vis_block_8 (Transformer Block) | 7.1M | 49 × 768 |
| 22 | changed (embedDim, numHeads, ffDim) | clip_block_16 (Transformer Block) | 13M | 576 × 1024 | txt_block_8 (Transformer Block) | 3.2M | 77 × 512 |
| 23 | changed (embedDim, numHeads, ffDim) | clip_block_17 (Transformer Block) | 13M | 576 × 1024 | vis_block_9 (Transformer Block) | 7.1M | 49 × 768 |
| 24 | changed (embedDim, numHeads, ffDim) | clip_block_18 (Transformer Block) | 13M | 576 × 1024 | txt_block_9 (Transformer Block) | 3.2M | 77 × 512 |
| 25 | changed (embedDim, numHeads, ffDim) | clip_block_19 (Transformer Block) | 13M | 576 × 1024 | vis_block_10 (Transformer Block) | 7.1M | 49 × 768 |
| 26 | changed (embedDim, numHeads, ffDim) | clip_block_20 (Transformer Block) | 13M | 576 × 1024 | txt_block_10 (Transformer Block) | 3.2M | 77 × 512 |
| 27 | changed (embedDim, numHeads, ffDim) | clip_block_21 (Transformer Block) | 13M | 576 × 1024 | vis_block_11 (Transformer Block) | 7.1M | 49 × 768 |
| 28 | changed (embedDim, numHeads, ffDim) | clip_block_22 (Transformer Block) | 13M | 576 × 1024 | txt_block_11 (Transformer Block) | 3.2M | 77 × 512 |
| 29 | changed (embedDim, numHeads, ffDim) | clip_block_23 (Transformer Block) | 13M | 576 × 1024 | vis_block_12 (Transformer Block) | 7.1M | 49 × 768 |
| 30 | changed (embedDim, numHeads, ffDim) | clip_block_24 (Transformer Block) | 13M | 576 × 1024 | txt_block_12 (Transformer Block) | 3.2M | 77 × 512 |
| 31 | removed | projector_1 (Linear) | 4.2M | 576 × 4096 |  | |  |
| 32 | removed | projector_gelu (Gelu) |  | 576 × 4096 |  | |  |
| 33 | removed | projector_2 (Linear) | 17M | 576 × 4096 |  | |  |
| 34 | removed | vision_tokens (Reshape) |  | 1 × 576 × 4096 |  | |  |
| 35 | removed | image+text_tokens (Concatenate) |  | 1 × 2624 × 4096 |  | |  |
| 36 | removed | llm_attn_norm_1 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 37 | removed | llm_attn_1 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 38 | removed | llm_res1_1 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 39 | removed | llm_ffn_norm_1 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 40 | removed | llm_ffn_1 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 41 | removed | llm_res2_1 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 42 | removed | llm_attn_norm_2 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 43 | removed | llm_attn_2 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 44 | removed | llm_res1_2 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 45 | removed | llm_ffn_norm_2 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 46 | removed | llm_ffn_2 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 47 | removed | llm_res2_2 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 48 | removed | llm_attn_norm_3 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 49 | removed | llm_attn_3 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 50 | removed | llm_res1_3 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 51 | removed | llm_ffn_norm_3 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 52 | removed | llm_ffn_3 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 53 | removed | llm_res2_3 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 54 | removed | llm_attn_norm_4 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 55 | removed | llm_attn_4 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 56 | removed | llm_res1_4 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 57 | removed | llm_ffn_norm_4 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 58 | removed | llm_ffn_4 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 59 | removed | llm_res2_4 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 60 | removed | llm_attn_norm_5 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 61 | removed | llm_attn_5 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 62 | removed | llm_res1_5 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 63 | removed | llm_ffn_norm_5 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 64 | removed | llm_ffn_5 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 65 | removed | llm_res2_5 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 66 | removed | llm_attn_norm_6 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 67 | removed | llm_attn_6 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 68 | removed | llm_res1_6 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 69 | removed | llm_ffn_norm_6 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 70 | removed | llm_ffn_6 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 71 | removed | llm_res2_6 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 72 | removed | llm_attn_norm_7 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 73 | removed | llm_attn_7 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 74 | removed | llm_res1_7 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 75 | removed | llm_ffn_norm_7 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 76 | removed | llm_ffn_7 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 77 | removed | llm_res2_7 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 78 | removed | llm_attn_norm_8 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 79 | removed | llm_attn_8 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 80 | removed | llm_res1_8 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 81 | removed | llm_ffn_norm_8 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 82 | removed | llm_ffn_8 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 83 | removed | llm_res2_8 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 84 | removed | llm_attn_norm_9 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 85 | removed | llm_attn_9 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 86 | removed | llm_res1_9 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 87 | removed | llm_ffn_norm_9 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 88 | removed | llm_ffn_9 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 89 | removed | llm_res2_9 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 90 | removed | llm_attn_norm_10 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 91 | removed | llm_attn_10 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 92 | removed | llm_res1_10 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 93 | removed | llm_ffn_norm_10 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 94 | removed | llm_ffn_10 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 95 | removed | llm_res2_10 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 96 | removed | llm_attn_norm_11 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 97 | removed | llm_attn_11 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 98 | removed | llm_res1_11 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 99 | removed | llm_ffn_norm_11 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 100 | removed | llm_ffn_11 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 101 | removed | llm_res2_11 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 102 | removed | llm_attn_norm_12 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 103 | removed | llm_attn_12 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 104 | removed | llm_res1_12 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 105 | removed | llm_ffn_norm_12 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 106 | removed | llm_ffn_12 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 107 | removed | llm_res2_12 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 108 | removed | llm_attn_norm_13 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 109 | removed | llm_attn_13 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 110 | removed | llm_res1_13 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 111 | removed | llm_ffn_norm_13 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 112 | removed | llm_ffn_13 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 113 | removed | llm_res2_13 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 114 | removed | llm_attn_norm_14 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 115 | removed | llm_attn_14 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 116 | removed | llm_res1_14 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 117 | removed | llm_ffn_norm_14 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 118 | removed | llm_ffn_14 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 119 | removed | llm_res2_14 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 120 | removed | llm_attn_norm_15 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 121 | removed | llm_attn_15 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 122 | removed | llm_res1_15 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 123 | removed | llm_ffn_norm_15 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 124 | removed | llm_ffn_15 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 125 | removed | llm_res2_15 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 126 | removed | llm_attn_norm_16 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 127 | removed | llm_attn_16 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 128 | removed | llm_res1_16 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 129 | removed | llm_ffn_norm_16 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 130 | removed | llm_ffn_16 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 131 | removed | llm_res2_16 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 132 | removed | llm_attn_norm_17 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 133 | removed | llm_attn_17 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 134 | removed | llm_res1_17 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 135 | removed | llm_ffn_norm_17 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 136 | removed | llm_ffn_17 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 137 | removed | llm_res2_17 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 138 | removed | llm_attn_norm_18 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 139 | removed | llm_attn_18 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 140 | removed | llm_res1_18 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 141 | removed | llm_ffn_norm_18 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 142 | removed | llm_ffn_18 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 143 | removed | llm_res2_18 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 144 | removed | llm_attn_norm_19 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 145 | removed | llm_attn_19 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 146 | removed | llm_res1_19 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 147 | removed | llm_ffn_norm_19 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 148 | removed | llm_ffn_19 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 149 | removed | llm_res2_19 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 150 | removed | llm_attn_norm_20 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 151 | removed | llm_attn_20 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 152 | removed | llm_res1_20 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 153 | removed | llm_ffn_norm_20 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 154 | removed | llm_ffn_20 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 155 | removed | llm_res2_20 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 156 | removed | llm_attn_norm_21 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 157 | removed | llm_attn_21 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 158 | removed | llm_res1_21 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 159 | removed | llm_ffn_norm_21 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 160 | removed | llm_ffn_21 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 161 | removed | llm_res2_21 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 162 | removed | llm_attn_norm_22 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 163 | removed | llm_attn_22 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 164 | removed | llm_res1_22 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 165 | removed | llm_ffn_norm_22 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 166 | removed | llm_ffn_22 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 167 | removed | llm_res2_22 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 168 | removed | llm_attn_norm_23 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 169 | removed | llm_attn_23 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 170 | removed | llm_res1_23 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 171 | removed | llm_ffn_norm_23 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 172 | removed | llm_ffn_23 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 173 | removed | llm_res2_23 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 174 | removed | llm_attn_norm_24 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 175 | removed | llm_attn_24 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 176 | removed | llm_res1_24 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 177 | removed | llm_ffn_norm_24 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 178 | removed | llm_ffn_24 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 179 | removed | llm_res2_24 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 180 | removed | llm_attn_norm_25 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 181 | removed | llm_attn_25 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 182 | removed | llm_res1_25 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 183 | removed | llm_ffn_norm_25 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 184 | removed | llm_ffn_25 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 185 | removed | llm_res2_25 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 186 | removed | llm_attn_norm_26 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 187 | removed | llm_attn_26 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 188 | removed | llm_res1_26 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 189 | removed | llm_ffn_norm_26 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 190 | removed | llm_ffn_26 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 191 | removed | llm_res2_26 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 192 | removed | llm_attn_norm_27 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 193 | removed | llm_attn_27 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 194 | removed | llm_res1_27 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 195 | removed | llm_ffn_norm_27 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 196 | removed | llm_ffn_27 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 197 | removed | llm_res2_27 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 198 | removed | llm_attn_norm_28 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 199 | removed | llm_attn_28 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 200 | removed | llm_res1_28 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 201 | removed | llm_ffn_norm_28 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 202 | removed | llm_ffn_28 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 203 | removed | llm_res2_28 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 204 | removed | llm_attn_norm_29 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 205 | removed | llm_attn_29 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 206 | removed | llm_res1_29 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 207 | removed | llm_ffn_norm_29 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 208 | removed | llm_ffn_29 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 209 | removed | llm_res2_29 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 210 | removed | llm_attn_norm_30 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 211 | removed | llm_attn_30 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 212 | removed | llm_res1_30 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 213 | removed | llm_ffn_norm_30 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 214 | removed | llm_ffn_30 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 215 | removed | llm_res2_30 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 216 | removed | llm_attn_norm_31 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 217 | removed | llm_attn_31 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 218 | removed | llm_res1_31 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 219 | removed | llm_ffn_norm_31 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 220 | removed | llm_ffn_31 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 221 | removed | llm_res2_31 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 222 | removed | llm_attn_norm_32 (Rms Norm) | 4.1K | 1 × 2624 × 4096 |  | |  |
| 223 | removed | llm_attn_32 (Grouped Query Attention) | 67M | 1 × 2624 × 4096 |  | |  |
| 224 | removed | llm_res1_32 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 225 | changed (type, normalizedShape) | llm_ffn_norm_32 (Rms Norm) | 4.1K | 1 × 2624 × 4096 | vis_norm (Layer Norm) | 1.5K | 49 × 768 |
| 226 | removed | llm_ffn_32 (Swiglu) | 135M | 1 × 2624 × 4096 |  | |  |
| 227 | removed | llm_res2_32 (Add) |  | 1 × 2624 × 4096 |  | |  |
| 228 | changed (type, normalizedShape) | final_norm (Rms Norm) | 4.1K | 1 × 2624 × 4096 | txt_norm (Layer Norm) | 1.0K | 77 × 512 |
| 229 | added |  | |  | to_channels_vis (Permute) |  | 768 × 49 |
| 230 | added |  | |  | to_channels_txt (Permute) |  | 512 × 77 |
| 231 | added |  | |  | vis_pool (Global Avg Pool1d) |  | 768 |
| 232 | added |  | |  | txt_pool (Global Avg Pool1d) |  | 512 |
| 233 | added |  | |  | img_proj (Linear) | 394K | 512 |
| 234 | changed (inFeatures, outFeatures, bias) | lm_head (Linear) | 131M | 1 × 2624 × 32000 | txt_proj (Linear) | 263K | 512 |
| 235 | added |  | |  | similarity (Matmul) |  | 512 |
| 236 | same | logits (Output) |  | 1 × 2624 × 32000 | logits (Output) |  | 512 |

## What this is not

- The two are priced at different declared inputs (3 × 336 × 336 against 3 × 224 × 224), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

## Graphs

- LLaVA-1.5: https://neurarch.com/templates/llava-1.5-7b/model.json
- CLIP ViT-B/32: https://neurarch.com/templates/clip-vit-b32/model.json
- Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`