# Llama-4 Scout

> 109B natively-multimodal MoE LLM — interleaved dense/MoE layers (16 experts) with iRoPE (interleaved no-RoPE) for long context (Meta 2025)

Reference the newest open MoE decoder: alternating dense and expert-routed blocks. A strong base for studying sparse-activation LLMs.

- Category: NLP/LLM
- Layers: 434
- Parameters: 102.55B
- Input shape (batchless): 1 × 10485760
- Output shape: 1 × 10486336 × 202048
- Verifier verdict: warn
- Graph JSON: https://neurarch.com/templates/llama-4-scout/model.json
- Open on the canvas: https://neurarch.com/?template=llama-4-scout

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | Input | Input | shape=[1, 10485760] | 1 × 10485760 |
| 2 | Embedding | Embedding | vocabSize=202048 | 1 × 10485760 × 5120 |
| 3 | Vision input | Input | shape=[3, 336, 336] | 3 × 336 × 336 |
| 4 | PatchEmbed | Patch Embed | embedDim=1408, patchSize=14 | 576 × 1408 |
| 5 | Vision_LN_1 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 6 | Vision_Attn_1 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 7 | Vision_Add_1 | Add |  | 576 × 1408 |
| 8 | Vision_FFN_1 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 9 | Vision_LN_2 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 10 | Vision_Attn_2 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 11 | Vision_Add_2 | Add |  | 576 × 1408 |
| 12 | Vision_FFN_2 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 13 | Vision_LN_3 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 14 | Vision_Attn_3 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 15 | Vision_Add_3 | Add |  | 576 × 1408 |
| 16 | Vision_FFN_3 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 17 | Vision_LN_4 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 18 | Vision_Attn_4 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 19 | Vision_Add_4 | Add |  | 576 × 1408 |
| 20 | Vision_FFN_4 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 21 | Vision_LN_5 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 22 | Vision_Attn_5 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 23 | Vision_Add_5 | Add |  | 576 × 1408 |
| 24 | Vision_FFN_5 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 25 | Vision_LN_6 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 26 | Vision_Attn_6 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 27 | Vision_Add_6 | Add |  | 576 × 1408 |
| 28 | Vision_FFN_6 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 29 | Vision_LN_7 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 30 | Vision_Attn_7 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 31 | Vision_Add_7 | Add |  | 576 × 1408 |
| 32 | Vision_FFN_7 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 33 | Vision_LN_8 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 34 | Vision_Attn_8 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 35 | Vision_Add_8 | Add |  | 576 × 1408 |
| 36 | Vision_FFN_8 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 37 | Vision_LN_9 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 38 | Vision_Attn_9 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 39 | Vision_Add_9 | Add |  | 576 × 1408 |
| 40 | Vision_FFN_9 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 41 | Vision_LN_10 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 42 | Vision_Attn_10 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 43 | Vision_Add_10 | Add |  | 576 × 1408 |
| 44 | Vision_FFN_10 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 45 | Vision_LN_11 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 46 | Vision_Attn_11 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 47 | Vision_Add_11 | Add |  | 576 × 1408 |
| 48 | Vision_FFN_11 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 49 | Vision_LN_12 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 50 | Vision_Attn_12 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 51 | Vision_Add_12 | Add |  | 576 × 1408 |
| 52 | Vision_FFN_12 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 53 | Vision_LN_13 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 54 | Vision_Attn_13 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 55 | Vision_Add_13 | Add |  | 576 × 1408 |
| 56 | Vision_FFN_13 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 57 | Vision_LN_14 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 58 | Vision_Attn_14 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 59 | Vision_Add_14 | Add |  | 576 × 1408 |
| 60 | Vision_FFN_14 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 61 | Vision_LN_15 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 62 | Vision_Attn_15 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 63 | Vision_Add_15 | Add |  | 576 × 1408 |
| 64 | Vision_FFN_15 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 65 | Vision_LN_16 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 66 | Vision_Attn_16 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 67 | Vision_Add_16 | Add |  | 576 × 1408 |
| 68 | Vision_FFN_16 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 69 | Vision_LN_17 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 70 | Vision_Attn_17 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 71 | Vision_Add_17 | Add |  | 576 × 1408 |
| 72 | Vision_FFN_17 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 73 | Vision_LN_18 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 74 | Vision_Attn_18 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 75 | Vision_Add_18 | Add |  | 576 × 1408 |
| 76 | Vision_FFN_18 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 77 | Vision_LN_19 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 78 | Vision_Attn_19 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 79 | Vision_Add_19 | Add |  | 576 × 1408 |
| 80 | Vision_FFN_19 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 81 | Vision_LN_20 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 82 | Vision_Attn_20 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 83 | Vision_Add_20 | Add |  | 576 × 1408 |
| 84 | Vision_FFN_20 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 85 | Vision_LN_21 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 86 | Vision_Attn_21 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 87 | Vision_Add_21 | Add |  | 576 × 1408 |
| 88 | Vision_FFN_21 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 89 | Vision_LN_22 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 90 | Vision_Attn_22 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 91 | Vision_Add_22 | Add |  | 576 × 1408 |
| 92 | Vision_FFN_22 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 93 | Vision_LN_23 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 94 | Vision_Attn_23 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 95 | Vision_Add_23 | Add |  | 576 × 1408 |
| 96 | Vision_FFN_23 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 97 | Vision_LN_24 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 98 | Vision_Attn_24 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 99 | Vision_Add_24 | Add |  | 576 × 1408 |
| 100 | Vision_FFN_24 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 101 | Vision_LN_25 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 102 | Vision_Attn_25 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 103 | Vision_Add_25 | Add |  | 576 × 1408 |
| 104 | Vision_FFN_25 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 105 | Vision_LN_26 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 106 | Vision_Attn_26 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 107 | Vision_Add_26 | Add |  | 576 × 1408 |
| 108 | Vision_FFN_26 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 109 | Vision_LN_27 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 110 | Vision_Attn_27 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 111 | Vision_Add_27 | Add |  | 576 × 1408 |
| 112 | Vision_FFN_27 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 113 | Vision_LN_28 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 114 | Vision_Attn_28 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 115 | Vision_Add_28 | Add |  | 576 × 1408 |
| 116 | Vision_FFN_28 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 117 | Vision_LN_29 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 118 | Vision_Attn_29 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 119 | Vision_Add_29 | Add |  | 576 × 1408 |
| 120 | Vision_FFN_29 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 121 | Vision_LN_30 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 122 | Vision_Attn_30 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 123 | Vision_Add_30 | Add |  | 576 × 1408 |
| 124 | Vision_FFN_30 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 125 | Vision_LN_31 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 126 | Vision_Attn_31 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 127 | Vision_Add_31 | Add |  | 576 × 1408 |
| 128 | Vision_FFN_31 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 129 | Vision_LN_32 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 130 | Vision_Attn_32 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 131 | Vision_Add_32 | Add |  | 576 × 1408 |
| 132 | Vision_FFN_32 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 133 | Vision_LN_33 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 134 | Vision_Attn_33 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 135 | Vision_Add_33 | Add |  | 576 × 1408 |
| 136 | Vision_FFN_33 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 137 | Vision_LN_34 | LayerNorm | normalizedShape=1408 | 576 × 1408 |
| 138 | Vision_Attn_34 | Multi-Head Attention | numHeads=16 | 576 × 1408 |
| 139 | Vision_Add_34 | Add |  | 576 × 1408 |
| 140 | Vision_FFN_34 | Feed Forward | ffDim=5632 | 576 × 1408 |
| 141 | Vision projector | Projection |  | 576 × 5120 |
| 142 | Vision tokens | Reshape | shape=[1, 576, 5120] | 1 × 576 × 5120 |
| 143 | Multimodal fusion (concat tokens) | Concatenate |  | 1 × 10486336 × 5120 |
| 144 | RMSNorm_1_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 145 | Attention_1 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 146 | Add_1_attn | Add |  | 1 × 10486336 × 5120 |
| 147 | RMSNorm_1_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 148 | MoE_1 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 149 | Add_1_ffn | Add |  | 1 × 10486336 × 5120 |
| 150 | RMSNorm_2_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 151 | Attention_2 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 152 | Add_2_attn | Add |  | 1 × 10486336 × 5120 |
| 153 | RMSNorm_2_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 154 | MoE_2 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 155 | Add_2_ffn | Add |  | 1 × 10486336 × 5120 |
| 156 | RMSNorm_3_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 157 | Attention_3 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 158 | Add_3_attn | Add |  | 1 × 10486336 × 5120 |
| 159 | RMSNorm_3_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 160 | MoE_3 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 161 | Add_3_ffn | Add |  | 1 × 10486336 × 5120 |
| 162 | RMSNorm_4_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 163 | Attention_4 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 164 | Add_4_attn | Add |  | 1 × 10486336 × 5120 |
| 165 | RMSNorm_4_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 166 | MoE_4 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 167 | Add_4_ffn | Add |  | 1 × 10486336 × 5120 |
| 168 | RMSNorm_5_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 169 | Attention_5 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 170 | Add_5_attn | Add |  | 1 × 10486336 × 5120 |
| 171 | RMSNorm_5_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 172 | MoE_5 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 173 | Add_5_ffn | Add |  | 1 × 10486336 × 5120 |
| 174 | RMSNorm_6_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 175 | Attention_6 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 176 | Add_6_attn | Add |  | 1 × 10486336 × 5120 |
| 177 | RMSNorm_6_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 178 | MoE_6 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 179 | Add_6_ffn | Add |  | 1 × 10486336 × 5120 |
| 180 | RMSNorm_7_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 181 | Attention_7 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 182 | Add_7_attn | Add |  | 1 × 10486336 × 5120 |
| 183 | RMSNorm_7_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 184 | MoE_7 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 185 | Add_7_ffn | Add |  | 1 × 10486336 × 5120 |
| 186 | RMSNorm_8_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 187 | Attention_8 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 188 | Add_8_attn | Add |  | 1 × 10486336 × 5120 |
| 189 | RMSNorm_8_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 190 | MoE_8 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 191 | Add_8_ffn | Add |  | 1 × 10486336 × 5120 |
| 192 | RMSNorm_9_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 193 | Attention_9 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 194 | Add_9_attn | Add |  | 1 × 10486336 × 5120 |
| 195 | RMSNorm_9_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 196 | MoE_9 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 197 | Add_9_ffn | Add |  | 1 × 10486336 × 5120 |
| 198 | RMSNorm_10_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 199 | Attention_10 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 200 | Add_10_attn | Add |  | 1 × 10486336 × 5120 |
| 201 | RMSNorm_10_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 202 | MoE_10 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 203 | Add_10_ffn | Add |  | 1 × 10486336 × 5120 |
| 204 | RMSNorm_11_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 205 | Attention_11 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 206 | Add_11_attn | Add |  | 1 × 10486336 × 5120 |
| 207 | RMSNorm_11_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 208 | MoE_11 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 209 | Add_11_ffn | Add |  | 1 × 10486336 × 5120 |
| 210 | RMSNorm_12_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 211 | Attention_12 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 212 | Add_12_attn | Add |  | 1 × 10486336 × 5120 |
| 213 | RMSNorm_12_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 214 | MoE_12 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 215 | Add_12_ffn | Add |  | 1 × 10486336 × 5120 |
| 216 | RMSNorm_13_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 217 | Attention_13 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 218 | Add_13_attn | Add |  | 1 × 10486336 × 5120 |
| 219 | RMSNorm_13_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 220 | MoE_13 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 221 | Add_13_ffn | Add |  | 1 × 10486336 × 5120 |
| 222 | RMSNorm_14_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 223 | Attention_14 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 224 | Add_14_attn | Add |  | 1 × 10486336 × 5120 |
| 225 | RMSNorm_14_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 226 | MoE_14 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 227 | Add_14_ffn | Add |  | 1 × 10486336 × 5120 |
| 228 | RMSNorm_15_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 229 | Attention_15 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 230 | Add_15_attn | Add |  | 1 × 10486336 × 5120 |
| 231 | RMSNorm_15_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 232 | MoE_15 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 233 | Add_15_ffn | Add |  | 1 × 10486336 × 5120 |
| 234 | RMSNorm_16_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 235 | Attention_16 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 236 | Add_16_attn | Add |  | 1 × 10486336 × 5120 |
| 237 | RMSNorm_16_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 238 | MoE_16 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 239 | Add_16_ffn | Add |  | 1 × 10486336 × 5120 |
| 240 | RMSNorm_17_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 241 | Attention_17 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 242 | Add_17_attn | Add |  | 1 × 10486336 × 5120 |
| 243 | RMSNorm_17_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 244 | MoE_17 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 245 | Add_17_ffn | Add |  | 1 × 10486336 × 5120 |
| 246 | RMSNorm_18_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 247 | Attention_18 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 248 | Add_18_attn | Add |  | 1 × 10486336 × 5120 |
| 249 | RMSNorm_18_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 250 | MoE_18 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 251 | Add_18_ffn | Add |  | 1 × 10486336 × 5120 |
| 252 | RMSNorm_19_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 253 | Attention_19 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 254 | Add_19_attn | Add |  | 1 × 10486336 × 5120 |
| 255 | RMSNorm_19_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 256 | MoE_19 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 257 | Add_19_ffn | Add |  | 1 × 10486336 × 5120 |
| 258 | RMSNorm_20_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 259 | Attention_20 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 260 | Add_20_attn | Add |  | 1 × 10486336 × 5120 |
| 261 | RMSNorm_20_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 262 | MoE_20 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 263 | Add_20_ffn | Add |  | 1 × 10486336 × 5120 |
| 264 | RMSNorm_21_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 265 | Attention_21 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 266 | Add_21_attn | Add |  | 1 × 10486336 × 5120 |
| 267 | RMSNorm_21_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 268 | MoE_21 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 269 | Add_21_ffn | Add |  | 1 × 10486336 × 5120 |
| 270 | RMSNorm_22_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 271 | Attention_22 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 272 | Add_22_attn | Add |  | 1 × 10486336 × 5120 |
| 273 | RMSNorm_22_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 274 | MoE_22 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 275 | Add_22_ffn | Add |  | 1 × 10486336 × 5120 |
| 276 | RMSNorm_23_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 277 | Attention_23 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 278 | Add_23_attn | Add |  | 1 × 10486336 × 5120 |
| 279 | RMSNorm_23_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 280 | MoE_23 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 281 | Add_23_ffn | Add |  | 1 × 10486336 × 5120 |
| 282 | RMSNorm_24_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 283 | Attention_24 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 284 | Add_24_attn | Add |  | 1 × 10486336 × 5120 |
| 285 | RMSNorm_24_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 286 | MoE_24 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 287 | Add_24_ffn | Add |  | 1 × 10486336 × 5120 |
| 288 | RMSNorm_25_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 289 | Attention_25 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 290 | Add_25_attn | Add |  | 1 × 10486336 × 5120 |
| 291 | RMSNorm_25_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 292 | MoE_25 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 293 | Add_25_ffn | Add |  | 1 × 10486336 × 5120 |
| 294 | RMSNorm_26_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 295 | Attention_26 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 296 | Add_26_attn | Add |  | 1 × 10486336 × 5120 |
| 297 | RMSNorm_26_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 298 | MoE_26 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 299 | Add_26_ffn | Add |  | 1 × 10486336 × 5120 |
| 300 | RMSNorm_27_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 301 | Attention_27 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 302 | Add_27_attn | Add |  | 1 × 10486336 × 5120 |
| 303 | RMSNorm_27_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 304 | MoE_27 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 305 | Add_27_ffn | Add |  | 1 × 10486336 × 5120 |
| 306 | RMSNorm_28_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 307 | Attention_28 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 308 | Add_28_attn | Add |  | 1 × 10486336 × 5120 |
| 309 | RMSNorm_28_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 310 | MoE_28 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 311 | Add_28_ffn | Add |  | 1 × 10486336 × 5120 |
| 312 | RMSNorm_29_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 313 | Attention_29 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 314 | Add_29_attn | Add |  | 1 × 10486336 × 5120 |
| 315 | RMSNorm_29_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 316 | MoE_29 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 317 | Add_29_ffn | Add |  | 1 × 10486336 × 5120 |
| 318 | RMSNorm_30_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 319 | Attention_30 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 320 | Add_30_attn | Add |  | 1 × 10486336 × 5120 |
| 321 | RMSNorm_30_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 322 | MoE_30 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 323 | Add_30_ffn | Add |  | 1 × 10486336 × 5120 |
| 324 | RMSNorm_31_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 325 | Attention_31 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 326 | Add_31_attn | Add |  | 1 × 10486336 × 5120 |
| 327 | RMSNorm_31_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 328 | MoE_31 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 329 | Add_31_ffn | Add |  | 1 × 10486336 × 5120 |
| 330 | RMSNorm_32_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 331 | Attention_32 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 332 | Add_32_attn | Add |  | 1 × 10486336 × 5120 |
| 333 | RMSNorm_32_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 334 | MoE_32 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 335 | Add_32_ffn | Add |  | 1 × 10486336 × 5120 |
| 336 | RMSNorm_33_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 337 | Attention_33 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 338 | Add_33_attn | Add |  | 1 × 10486336 × 5120 |
| 339 | RMSNorm_33_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 340 | MoE_33 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 341 | Add_33_ffn | Add |  | 1 × 10486336 × 5120 |
| 342 | RMSNorm_34_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 343 | Attention_34 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 344 | Add_34_attn | Add |  | 1 × 10486336 × 5120 |
| 345 | RMSNorm_34_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 346 | MoE_34 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 347 | Add_34_ffn | Add |  | 1 × 10486336 × 5120 |
| 348 | RMSNorm_35_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 349 | Attention_35 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 350 | Add_35_attn | Add |  | 1 × 10486336 × 5120 |
| 351 | RMSNorm_35_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 352 | MoE_35 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 353 | Add_35_ffn | Add |  | 1 × 10486336 × 5120 |
| 354 | RMSNorm_36_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 355 | Attention_36 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 356 | Add_36_attn | Add |  | 1 × 10486336 × 5120 |
| 357 | RMSNorm_36_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 358 | MoE_36 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 359 | Add_36_ffn | Add |  | 1 × 10486336 × 5120 |
| 360 | RMSNorm_37_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 361 | Attention_37 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 362 | Add_37_attn | Add |  | 1 × 10486336 × 5120 |
| 363 | RMSNorm_37_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 364 | MoE_37 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 365 | Add_37_ffn | Add |  | 1 × 10486336 × 5120 |
| 366 | RMSNorm_38_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 367 | Attention_38 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 368 | Add_38_attn | Add |  | 1 × 10486336 × 5120 |
| 369 | RMSNorm_38_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 370 | MoE_38 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 371 | Add_38_ffn | Add |  | 1 × 10486336 × 5120 |
| 372 | RMSNorm_39_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 373 | Attention_39 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 374 | Add_39_attn | Add |  | 1 × 10486336 × 5120 |
| 375 | RMSNorm_39_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 376 | MoE_39 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 377 | Add_39_ffn | Add |  | 1 × 10486336 × 5120 |
| 378 | RMSNorm_40_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 379 | Attention_40 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 380 | Add_40_attn | Add |  | 1 × 10486336 × 5120 |
| 381 | RMSNorm_40_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 382 | MoE_40 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 383 | Add_40_ffn | Add |  | 1 × 10486336 × 5120 |
| 384 | RMSNorm_41_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 385 | Attention_41 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 386 | Add_41_attn | Add |  | 1 × 10486336 × 5120 |
| 387 | RMSNorm_41_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 388 | MoE_41 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 389 | Add_41_ffn | Add |  | 1 × 10486336 × 5120 |
| 390 | RMSNorm_42_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 391 | Attention_42 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 392 | Add_42_attn | Add |  | 1 × 10486336 × 5120 |
| 393 | RMSNorm_42_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 394 | MoE_42 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 395 | Add_42_ffn | Add |  | 1 × 10486336 × 5120 |
| 396 | RMSNorm_43_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 397 | Attention_43 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 398 | Add_43_attn | Add |  | 1 × 10486336 × 5120 |
| 399 | RMSNorm_43_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 400 | MoE_43 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 401 | Add_43_ffn | Add |  | 1 × 10486336 × 5120 |
| 402 | RMSNorm_44_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 403 | Attention_44 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 404 | Add_44_attn | Add |  | 1 × 10486336 × 5120 |
| 405 | RMSNorm_44_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 406 | MoE_44 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 407 | Add_44_ffn | Add |  | 1 × 10486336 × 5120 |
| 408 | RMSNorm_45_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 409 | Attention_45 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 410 | Add_45_attn | Add |  | 1 × 10486336 × 5120 |
| 411 | RMSNorm_45_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 412 | MoE_45 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 413 | Add_45_ffn | Add |  | 1 × 10486336 × 5120 |
| 414 | RMSNorm_46_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 415 | Attention_46 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 416 | Add_46_attn | Add |  | 1 × 10486336 × 5120 |
| 417 | RMSNorm_46_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 418 | MoE_46 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 419 | Add_46_ffn | Add |  | 1 × 10486336 × 5120 |
| 420 | RMSNorm_47_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 421 | Attention_47 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 422 | Add_47_attn | Add |  | 1 × 10486336 × 5120 |
| 423 | RMSNorm_47_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 424 | MoE_47 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 425 | Add_47_ffn | Add |  | 1 × 10486336 × 5120 |
| 426 | RMSNorm_48_1 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 427 | Attention_48 | Grouped Query Attn | embedDim=5120, numHeads=40, numKVHeads=8 | 1 × 10486336 × 5120 |
| 428 | Add_48_attn | Add |  | 1 × 10486336 × 5120 |
| 429 | RMSNorm_48_2 | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 430 | MoE_48 | MoE Layer | embedDim=5120, numExperts=16, topK=1 | 1 × 10486336 × 5120 |
| 431 | Add_48_ffn | Add |  | 1 × 10486336 × 5120 |
| 432 | Final_RMSNorm | RMSNorm | normalizedShape=5120 | 1 × 10486336 × 5120 |
| 433 | LM_Head | Linear | outFeatures=202048, inFeatures=5120 | 1 × 10486336 × 202048 |
| 434 | Output | Output |  | 1 × 10486336 × 202048 |

## Verifier findings

- **warn** `attention-no-pe` at `Vision_Attn_1`: 82 attention layer(s) present but no positional encoding found. Attention is permutation-invariant, without position information the model cannot distinguish token order. Fix: Add a PositionalEncoding (sinusoidal) or RoPE layer before the first attention layer.
- **info** `moe-no-aux-loss` at `MoE_1`: MoE layers require an auxiliary router z-loss + load-balance loss during training to prevent expert collapse. This is not visible in the architecture diagram but must be in the training loop. Applies to all 48: MoE_1, MoE_2, MoE_3, MoE_4, MoE_5, MoE_6, MoE_7, MoE_8, MoE_9, MoE_10, MoE_11, MoE_12, MoE_13, MoE_14, MoE_15, MoE_16, MoE_17, MoE_18, MoE_19, MoE_20, MoE_21, MoE_22, MoE_23, MoE_24, MoE_25, MoE_26, MoE_27, MoE_28, MoE_29, MoE_30, MoE_31, MoE_32, MoE_33, MoE_34, MoE_35, MoE_36, MoE_37, MoE_38, MoE_39, MoE_40, MoE_41, MoE_42, MoE_43, MoE_44, MoE_45, MoE_46, MoE_47, MoE_48. Fix: Add a note on these layers. Typical aux_loss coefficient: 1e-2 (Mixtral/Switch Transformer).
- **warn** `huge-linear-params` at `LM_Head`: Linear "LM_Head" is 5120 × 202048 = 1034M parameters (~3.9 GB float32). A single dense layer this large usually means a feature map was flattened without pooling first; embedding / vocab-projection heads are the expected exception. Fix: Add a Global Average Pool or more downsampling before the Linear, or factorize it (low-rank / bottleneck projection).
- **info** `deep-attention-default-init` at `Vision_Attn_1`: At 82 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class Llama4Scout(nn.Module):
    def __init__(self):
        super().__init__()

        self.embedding_1 = nn.Embedding(202048, 5120)
        self.patchEmbed_1 = nn.Conv2d(3, 1408, kernel_size=14, stride=14)  # Patch embedding (ViT-style)
        self.layerNorm_1 = nn.LayerNorm(1408)
        self.multiHeadAttention_1 = nn.MultiheadAttention(embed_dim=1408, num_heads=16, batch_first=True)
        self.feedForward_1 = nn.Sequential(
            nn.Linear(1408, 5632),
            nn.ReLU(),
            nn.Linear(5632, 1408)
        )
        self.layerNorm_2 = nn.LayerNorm(1408)
        self.multiHeadAttention_2 = nn.MultiheadAttention(embed_dim=1408, num_heads=16, batch_first=True)
        self.feedForward_2 = nn.Sequential(
            nn.Linear(1408, 5632),
            nn.ReLU(),
            nn.Linear(5632, 1408)
        )
        self.layerNorm_3 = nn.LayerNorm(1408)
        self.multiHeadAttention_3 = nn.MultiheadAttention(embed_dim=1408, num_heads=16, batch_first=True)
        self.feedForward_3 = nn.Sequential(
            nn.Linear(1408, 5632),
            nn.ReLU(),
            nn.Linear(5632, 1408)
        )
        self.layerNorm_4 = nn.LayerNorm(1408)
        self.multiHeadAttention_4 = nn.MultiheadAttention(embed_dim=1408, num_heads=16, batch_first=True)
        self.feedForward_4 = nn.Sequential(
            nn.Linear(1408, 5632),
            nn.ReLU(),
            nn.Linear(5632, 1408)
        )
        self.layerNorm_5 = nn.LayerNorm(1408)
        self.multiHeadAttention_5 = nn.MultiheadAttention(embed_dim=1408, num_heads=16, batch_first=True)
        self.feedForward_5 = nn.Sequential(
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md