# LLaVA-1.5

> Vision-language model — CLIP image encoder + MLP projector feed visual tokens into a LLaMA decoder (Liu 2023)

Pick to see the canonical VLM recipe: a frozen vision encoder bridged into an LLM by a small projector. Foundation for image chat / VQA.

- Category: Multimodal
- Layers: 229
- Parameters: 7.06B
- Input shape (batchless): 3 × 336 × 336
- Output shape: 1 × 2624 × 32000
- Verifier verdict: warn
- Graph JSON: https://neurarch.com/templates/llava-1.5-7b/model.json
- Open on the canvas: https://neurarch.com/?template=llava-1.5-7b

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | image | Input | shape=[3, 336, 336] | 3 × 336 × 336 |
| 2 | clip_patch_14 | Patch Embed | embedDim=1024, patchSize=14 | 576 × 1024 |
| 3 | vis_pos | Positional Encoding | embedDim=1024, maxLen=576 | 576 × 1024 |
| 4 | clip_block_1 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 5 | clip_block_2 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 6 | clip_block_3 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 7 | clip_block_4 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 8 | clip_block_5 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 9 | clip_block_6 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 10 | clip_block_7 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 11 | clip_block_8 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 12 | clip_block_9 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 13 | clip_block_10 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 14 | clip_block_11 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 15 | clip_block_12 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 16 | clip_block_13 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 17 | clip_block_14 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 18 | clip_block_15 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 19 | clip_block_16 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 20 | clip_block_17 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 21 | clip_block_18 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 22 | clip_block_19 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 23 | clip_block_20 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 24 | clip_block_21 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 25 | clip_block_22 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 26 | clip_block_23 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 27 | clip_block_24 | Transformer Block | embedDim=1024, numHeads=16, ffDim=4096 | 576 × 1024 |
| 28 | projector_1 | Linear | outFeatures=4096, inFeatures=1024 | 576 × 4096 |
| 29 | projector_gelu | GELU |  | 576 × 4096 |
| 30 | projector_2 | Linear | outFeatures=4096, inFeatures=4096 | 576 × 4096 |
| 31 | vision_tokens | Reshape | shape=[1, 576, 4096] | 1 × 576 × 4096 |
| 32 | text_tokens | Input | shape=[1, 2048] | 1 × 2048 |
| 33 | token_embed | Embedding |  | 1 × 2048 × 4096 |
| 34 | image+text_tokens | Concatenate |  | 1 × 2624 × 4096 |
| 35 | llm_attn_norm_1 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 36 | llm_attn_1 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 37 | llm_res1_1 | Add |  | 1 × 2624 × 4096 |
| 38 | llm_ffn_norm_1 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 39 | llm_ffn_1 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 40 | llm_res2_1 | Add |  | 1 × 2624 × 4096 |
| 41 | llm_attn_norm_2 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 42 | llm_attn_2 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 43 | llm_res1_2 | Add |  | 1 × 2624 × 4096 |
| 44 | llm_ffn_norm_2 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 45 | llm_ffn_2 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 46 | llm_res2_2 | Add |  | 1 × 2624 × 4096 |
| 47 | llm_attn_norm_3 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 48 | llm_attn_3 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 49 | llm_res1_3 | Add |  | 1 × 2624 × 4096 |
| 50 | llm_ffn_norm_3 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 51 | llm_ffn_3 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 52 | llm_res2_3 | Add |  | 1 × 2624 × 4096 |
| 53 | llm_attn_norm_4 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 54 | llm_attn_4 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 55 | llm_res1_4 | Add |  | 1 × 2624 × 4096 |
| 56 | llm_ffn_norm_4 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 57 | llm_ffn_4 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 58 | llm_res2_4 | Add |  | 1 × 2624 × 4096 |
| 59 | llm_attn_norm_5 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 60 | llm_attn_5 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 61 | llm_res1_5 | Add |  | 1 × 2624 × 4096 |
| 62 | llm_ffn_norm_5 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 63 | llm_ffn_5 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 64 | llm_res2_5 | Add |  | 1 × 2624 × 4096 |
| 65 | llm_attn_norm_6 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 66 | llm_attn_6 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 67 | llm_res1_6 | Add |  | 1 × 2624 × 4096 |
| 68 | llm_ffn_norm_6 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 69 | llm_ffn_6 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 70 | llm_res2_6 | Add |  | 1 × 2624 × 4096 |
| 71 | llm_attn_norm_7 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 72 | llm_attn_7 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 73 | llm_res1_7 | Add |  | 1 × 2624 × 4096 |
| 74 | llm_ffn_norm_7 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 75 | llm_ffn_7 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 76 | llm_res2_7 | Add |  | 1 × 2624 × 4096 |
| 77 | llm_attn_norm_8 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 78 | llm_attn_8 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 79 | llm_res1_8 | Add |  | 1 × 2624 × 4096 |
| 80 | llm_ffn_norm_8 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 81 | llm_ffn_8 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 82 | llm_res2_8 | Add |  | 1 × 2624 × 4096 |
| 83 | llm_attn_norm_9 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 84 | llm_attn_9 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 85 | llm_res1_9 | Add |  | 1 × 2624 × 4096 |
| 86 | llm_ffn_norm_9 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 87 | llm_ffn_9 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 88 | llm_res2_9 | Add |  | 1 × 2624 × 4096 |
| 89 | llm_attn_norm_10 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 90 | llm_attn_10 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 91 | llm_res1_10 | Add |  | 1 × 2624 × 4096 |
| 92 | llm_ffn_norm_10 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 93 | llm_ffn_10 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 94 | llm_res2_10 | Add |  | 1 × 2624 × 4096 |
| 95 | llm_attn_norm_11 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 96 | llm_attn_11 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 97 | llm_res1_11 | Add |  | 1 × 2624 × 4096 |
| 98 | llm_ffn_norm_11 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 99 | llm_ffn_11 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 100 | llm_res2_11 | Add |  | 1 × 2624 × 4096 |
| 101 | llm_attn_norm_12 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 102 | llm_attn_12 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 103 | llm_res1_12 | Add |  | 1 × 2624 × 4096 |
| 104 | llm_ffn_norm_12 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 105 | llm_ffn_12 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 106 | llm_res2_12 | Add |  | 1 × 2624 × 4096 |
| 107 | llm_attn_norm_13 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 108 | llm_attn_13 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 109 | llm_res1_13 | Add |  | 1 × 2624 × 4096 |
| 110 | llm_ffn_norm_13 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 111 | llm_ffn_13 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 112 | llm_res2_13 | Add |  | 1 × 2624 × 4096 |
| 113 | llm_attn_norm_14 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 114 | llm_attn_14 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 115 | llm_res1_14 | Add |  | 1 × 2624 × 4096 |
| 116 | llm_ffn_norm_14 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 117 | llm_ffn_14 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 118 | llm_res2_14 | Add |  | 1 × 2624 × 4096 |
| 119 | llm_attn_norm_15 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 120 | llm_attn_15 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 121 | llm_res1_15 | Add |  | 1 × 2624 × 4096 |
| 122 | llm_ffn_norm_15 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 123 | llm_ffn_15 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 124 | llm_res2_15 | Add |  | 1 × 2624 × 4096 |
| 125 | llm_attn_norm_16 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 126 | llm_attn_16 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 127 | llm_res1_16 | Add |  | 1 × 2624 × 4096 |
| 128 | llm_ffn_norm_16 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 129 | llm_ffn_16 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 130 | llm_res2_16 | Add |  | 1 × 2624 × 4096 |
| 131 | llm_attn_norm_17 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 132 | llm_attn_17 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 133 | llm_res1_17 | Add |  | 1 × 2624 × 4096 |
| 134 | llm_ffn_norm_17 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 135 | llm_ffn_17 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 136 | llm_res2_17 | Add |  | 1 × 2624 × 4096 |
| 137 | llm_attn_norm_18 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 138 | llm_attn_18 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 139 | llm_res1_18 | Add |  | 1 × 2624 × 4096 |
| 140 | llm_ffn_norm_18 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 141 | llm_ffn_18 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 142 | llm_res2_18 | Add |  | 1 × 2624 × 4096 |
| 143 | llm_attn_norm_19 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 144 | llm_attn_19 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 145 | llm_res1_19 | Add |  | 1 × 2624 × 4096 |
| 146 | llm_ffn_norm_19 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 147 | llm_ffn_19 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 148 | llm_res2_19 | Add |  | 1 × 2624 × 4096 |
| 149 | llm_attn_norm_20 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 150 | llm_attn_20 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 151 | llm_res1_20 | Add |  | 1 × 2624 × 4096 |
| 152 | llm_ffn_norm_20 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 153 | llm_ffn_20 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 154 | llm_res2_20 | Add |  | 1 × 2624 × 4096 |
| 155 | llm_attn_norm_21 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 156 | llm_attn_21 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 157 | llm_res1_21 | Add |  | 1 × 2624 × 4096 |
| 158 | llm_ffn_norm_21 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 159 | llm_ffn_21 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 160 | llm_res2_21 | Add |  | 1 × 2624 × 4096 |
| 161 | llm_attn_norm_22 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 162 | llm_attn_22 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 163 | llm_res1_22 | Add |  | 1 × 2624 × 4096 |
| 164 | llm_ffn_norm_22 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 165 | llm_ffn_22 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 166 | llm_res2_22 | Add |  | 1 × 2624 × 4096 |
| 167 | llm_attn_norm_23 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 168 | llm_attn_23 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 169 | llm_res1_23 | Add |  | 1 × 2624 × 4096 |
| 170 | llm_ffn_norm_23 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 171 | llm_ffn_23 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 172 | llm_res2_23 | Add |  | 1 × 2624 × 4096 |
| 173 | llm_attn_norm_24 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 174 | llm_attn_24 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 175 | llm_res1_24 | Add |  | 1 × 2624 × 4096 |
| 176 | llm_ffn_norm_24 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 177 | llm_ffn_24 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 178 | llm_res2_24 | Add |  | 1 × 2624 × 4096 |
| 179 | llm_attn_norm_25 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 180 | llm_attn_25 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 181 | llm_res1_25 | Add |  | 1 × 2624 × 4096 |
| 182 | llm_ffn_norm_25 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 183 | llm_ffn_25 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 184 | llm_res2_25 | Add |  | 1 × 2624 × 4096 |
| 185 | llm_attn_norm_26 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 186 | llm_attn_26 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 187 | llm_res1_26 | Add |  | 1 × 2624 × 4096 |
| 188 | llm_ffn_norm_26 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 189 | llm_ffn_26 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 190 | llm_res2_26 | Add |  | 1 × 2624 × 4096 |
| 191 | llm_attn_norm_27 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 192 | llm_attn_27 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 193 | llm_res1_27 | Add |  | 1 × 2624 × 4096 |
| 194 | llm_ffn_norm_27 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 195 | llm_ffn_27 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 196 | llm_res2_27 | Add |  | 1 × 2624 × 4096 |
| 197 | llm_attn_norm_28 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 198 | llm_attn_28 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 199 | llm_res1_28 | Add |  | 1 × 2624 × 4096 |
| 200 | llm_ffn_norm_28 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 201 | llm_ffn_28 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 202 | llm_res2_28 | Add |  | 1 × 2624 × 4096 |
| 203 | llm_attn_norm_29 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 204 | llm_attn_29 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 205 | llm_res1_29 | Add |  | 1 × 2624 × 4096 |
| 206 | llm_ffn_norm_29 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 207 | llm_ffn_29 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 208 | llm_res2_29 | Add |  | 1 × 2624 × 4096 |
| 209 | llm_attn_norm_30 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 210 | llm_attn_30 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 211 | llm_res1_30 | Add |  | 1 × 2624 × 4096 |
| 212 | llm_ffn_norm_30 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 213 | llm_ffn_30 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 214 | llm_res2_30 | Add |  | 1 × 2624 × 4096 |
| 215 | llm_attn_norm_31 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 216 | llm_attn_31 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 217 | llm_res1_31 | Add |  | 1 × 2624 × 4096 |
| 218 | llm_ffn_norm_31 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 219 | llm_ffn_31 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 220 | llm_res2_31 | Add |  | 1 × 2624 × 4096 |
| 221 | llm_attn_norm_32 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 222 | llm_attn_32 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=32 | 1 × 2624 × 4096 |
| 223 | llm_res1_32 | Add |  | 1 × 2624 × 4096 |
| 224 | llm_ffn_norm_32 | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 225 | llm_ffn_32 | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2624 × 4096 |
| 226 | llm_res2_32 | Add |  | 1 × 2624 × 4096 |
| 227 | final_norm | RMSNorm | normalizedShape=4096 | 1 × 2624 × 4096 |
| 228 | lm_head | Linear | outFeatures=32000, inFeatures=4096 | 1 × 2624 × 32000 |
| 229 | logits | Output |  | 1 × 2624 × 32000 |

## Verifier findings

- **info** `full-mha-serving-cost` at `llm_attn_1`: 32 attention layers at embedDim 4096 cache full per-head K/V: about 512 KB per token at fp16, which dominates memory at long context. Grouped-query attention (e.g. 8:1) would cut this ~8×; multi-head latent attention (MLA) shrinks it ~10× or more. This is the move production LLMs make; it does not change the parameter count. Fix: Switch attention to groupedQueryAttention (set numKVHeads below numHeads, e.g. numHeads/4) or mla (a low-rank cached latent).
- **info** `deep-attention-default-init` at `llm_attn_1`: At 32 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))
- **warn** `kv-cache-context-budget` at `llm_attn_1`: Across 32 attention layers this design caches 512 KB per token, so a single 8,192-token sequence needs ~4.3 GB of KV cache before weights or activations. That exceeds the 4 GB budget this rule assumes for serving headroom. Fix: Cut KV width: raise the GQA ratio (fewer numKVHeads), switch to MLA, reduce depth or embedDim, or accept a shorter serving context.

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class LLaVA_15_7B(nn.Module):
    def __init__(self):
        super().__init__()

        self.patchEmbed_1 = nn.Conv2d(3, 1024, kernel_size=14, stride=14)  # Patch embedding (ViT-style)
        self.transformerBlock_1 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_2 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_3 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_4 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_5 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_6 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_7 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_8 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_9 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_10 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_11 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_12 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_13 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_14 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_15 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_16 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_17 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_18 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_19 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_20 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_21 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_22 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_23 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.transformerBlock_24 = nn.TransformerEncoderLayer(d_model=1024, nhead=16, dim_feedforward=4096, batch_first=True)
        self.linear_1 = nn.Linear(1024, 4096)
        self.gelu_1 = nn.GELU()
        self.linear_2 = nn.Linear(4096, 4096)
        self.embedding_1 = nn.Embedding(32000, 4096)
        self.rmsNorm_1 = nn.RMSNorm(4096)
        self.groupedQueryAttention_1 = nn.ModuleDict({
            'q_proj': nn.Linear(4096, 4096,        bias=False),   # 32 heads × 128
            'k_proj': nn.Linear(4096, 4096, bias=False),   # 32 KV heads × 128
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md