# Qwen3-8B vs Llama-4 Scout

Two 2026 open models with different attention budgets.

**Llama-4 Scout has 94B more parameters than Qwen3-8B: 249 layers added, 36 removed, 112 changed.**

Source: https://neurarch.com/diff/qwen3-8b-vs-llama-4-scout.html

## Sides

| | Qwen3-8B | Llama-4 Scout |
|---|---|---|
| Layers | 219 | 431 |
| Parameters | 8.2B | 103B |
| Input | 1 × 40960 | 1 × 10485760 |
| Output | 1 × 40960 × 151936 | 1 × 10486336 × 202048 |
| Forward-passes | yes | yes |
| Est. train cost | $20,863 | $1,015,410,581 |
| T4 16GB | no | no |
| A100 40GB | no | no |
| H100 80GB | no | no |

## Deltas (Llama-4 Scout relative to Qwen3-8B)

- Parameters: +94B (13× the size)
- Layers: +212
- Added 249, removed 36, changed 112, unchanged 73

## Layer by layer

| # | Status | Qwen3-8B | Params | Output | Llama-4 Scout | Params | Output |
|---|---|---|---|---|---|---|---|
| 1 | added |  | |  | Input (Input) |  | 1 × 10485760 |
| 2 | changed (shape) | Input (Input) |  | 1 × 40960 | Vision input (Input) |  | 3 × 336 × 336 |
| 3 | changed (vocabSize, embeddingDim, maxSeqLen) | Embedding (Embedding) | 622M | 1 × 40960 × 4096 | Embedding (Embedding) | 1.0B | 1 × 10485760 × 5120 |
| 4 | added |  | |  | PatchEmbed (Patch Embed) | 829K | 576 × 1408 |
| 5 | added |  | |  | Vision_LN_1 (Layer Norm) | 2.8K | 576 × 1408 |
| 6 | added |  | |  | Vision_Attn_1 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 7 | added |  | |  | Vision_Add_1 (Add) |  | 576 × 1408 |
| 8 | added |  | |  | Vision_FFN_1 (Feed Forward) | 16M | 576 × 1408 |
| 9 | added |  | |  | Vision_LN_2 (Layer Norm) | 2.8K | 576 × 1408 |
| 10 | added |  | |  | Vision_Attn_2 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 11 | added |  | |  | Vision_Add_2 (Add) |  | 576 × 1408 |
| 12 | added |  | |  | Vision_FFN_2 (Feed Forward) | 16M | 576 × 1408 |
| 13 | added |  | |  | Vision_LN_3 (Layer Norm) | 2.8K | 576 × 1408 |
| 14 | added |  | |  | Vision_Attn_3 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 15 | added |  | |  | Vision_Add_3 (Add) |  | 576 × 1408 |
| 16 | added |  | |  | Vision_FFN_3 (Feed Forward) | 16M | 576 × 1408 |
| 17 | added |  | |  | Vision_LN_4 (Layer Norm) | 2.8K | 576 × 1408 |
| 18 | added |  | |  | Vision_Attn_4 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 19 | added |  | |  | Vision_Add_4 (Add) |  | 576 × 1408 |
| 20 | added |  | |  | Vision_FFN_4 (Feed Forward) | 16M | 576 × 1408 |
| 21 | added |  | |  | Vision_LN_5 (Layer Norm) | 2.8K | 576 × 1408 |
| 22 | added |  | |  | Vision_Attn_5 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 23 | added |  | |  | Vision_Add_5 (Add) |  | 576 × 1408 |
| 24 | added |  | |  | Vision_FFN_5 (Feed Forward) | 16M | 576 × 1408 |
| 25 | added |  | |  | Vision_LN_6 (Layer Norm) | 2.8K | 576 × 1408 |
| 26 | added |  | |  | Vision_Attn_6 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 27 | added |  | |  | Vision_Add_6 (Add) |  | 576 × 1408 |
| 28 | added |  | |  | Vision_FFN_6 (Feed Forward) | 16M | 576 × 1408 |
| 29 | added |  | |  | Vision_LN_7 (Layer Norm) | 2.8K | 576 × 1408 |
| 30 | added |  | |  | Vision_Attn_7 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 31 | added |  | |  | Vision_Add_7 (Add) |  | 576 × 1408 |
| 32 | added |  | |  | Vision_FFN_7 (Feed Forward) | 16M | 576 × 1408 |
| 33 | added |  | |  | Vision_LN_8 (Layer Norm) | 2.8K | 576 × 1408 |
| 34 | added |  | |  | Vision_Attn_8 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 35 | added |  | |  | Vision_Add_8 (Add) |  | 576 × 1408 |
| 36 | added |  | |  | Vision_FFN_8 (Feed Forward) | 16M | 576 × 1408 |
| 37 | added |  | |  | Vision_LN_9 (Layer Norm) | 2.8K | 576 × 1408 |
| 38 | added |  | |  | Vision_Attn_9 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 39 | added |  | |  | Vision_Add_9 (Add) |  | 576 × 1408 |
| 40 | added |  | |  | Vision_FFN_9 (Feed Forward) | 16M | 576 × 1408 |
| 41 | added |  | |  | Vision_LN_10 (Layer Norm) | 2.8K | 576 × 1408 |
| 42 | added |  | |  | Vision_Attn_10 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 43 | added |  | |  | Vision_Add_10 (Add) |  | 576 × 1408 |
| 44 | added |  | |  | Vision_FFN_10 (Feed Forward) | 16M | 576 × 1408 |
| 45 | added |  | |  | Vision_LN_11 (Layer Norm) | 2.8K | 576 × 1408 |
| 46 | added |  | |  | Vision_Attn_11 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 47 | added |  | |  | Vision_Add_11 (Add) |  | 576 × 1408 |
| 48 | added |  | |  | Vision_FFN_11 (Feed Forward) | 16M | 576 × 1408 |
| 49 | added |  | |  | Vision_LN_12 (Layer Norm) | 2.8K | 576 × 1408 |
| 50 | added |  | |  | Vision_Attn_12 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 51 | added |  | |  | Vision_Add_12 (Add) |  | 576 × 1408 |
| 52 | added |  | |  | Vision_FFN_12 (Feed Forward) | 16M | 576 × 1408 |
| 53 | added |  | |  | Vision_LN_13 (Layer Norm) | 2.8K | 576 × 1408 |
| 54 | added |  | |  | Vision_Attn_13 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 55 | added |  | |  | Vision_Add_13 (Add) |  | 576 × 1408 |
| 56 | added |  | |  | Vision_FFN_13 (Feed Forward) | 16M | 576 × 1408 |
| 57 | added |  | |  | Vision_LN_14 (Layer Norm) | 2.8K | 576 × 1408 |
| 58 | added |  | |  | Vision_Attn_14 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 59 | added |  | |  | Vision_Add_14 (Add) |  | 576 × 1408 |
| 60 | added |  | |  | Vision_FFN_14 (Feed Forward) | 16M | 576 × 1408 |
| 61 | added |  | |  | Vision_LN_15 (Layer Norm) | 2.8K | 576 × 1408 |
| 62 | added |  | |  | Vision_Attn_15 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 63 | added |  | |  | Vision_Add_15 (Add) |  | 576 × 1408 |
| 64 | added |  | |  | Vision_FFN_15 (Feed Forward) | 16M | 576 × 1408 |
| 65 | added |  | |  | Vision_LN_16 (Layer Norm) | 2.8K | 576 × 1408 |
| 66 | added |  | |  | Vision_Attn_16 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 67 | added |  | |  | Vision_Add_16 (Add) |  | 576 × 1408 |
| 68 | added |  | |  | Vision_FFN_16 (Feed Forward) | 16M | 576 × 1408 |
| 69 | added |  | |  | Vision_LN_17 (Layer Norm) | 2.8K | 576 × 1408 |
| 70 | added |  | |  | Vision_Attn_17 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 71 | added |  | |  | Vision_Add_17 (Add) |  | 576 × 1408 |
| 72 | added |  | |  | Vision_FFN_17 (Feed Forward) | 16M | 576 × 1408 |
| 73 | added |  | |  | Vision_LN_18 (Layer Norm) | 2.8K | 576 × 1408 |
| 74 | added |  | |  | Vision_Attn_18 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 75 | added |  | |  | Vision_Add_18 (Add) |  | 576 × 1408 |
| 76 | added |  | |  | Vision_FFN_18 (Feed Forward) | 16M | 576 × 1408 |
| 77 | added |  | |  | Vision_LN_19 (Layer Norm) | 2.8K | 576 × 1408 |
| 78 | added |  | |  | Vision_Attn_19 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 79 | added |  | |  | Vision_Add_19 (Add) |  | 576 × 1408 |
| 80 | added |  | |  | Vision_FFN_19 (Feed Forward) | 16M | 576 × 1408 |
| 81 | added |  | |  | Vision_LN_20 (Layer Norm) | 2.8K | 576 × 1408 |
| 82 | added |  | |  | Vision_Attn_20 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 83 | added |  | |  | Vision_Add_20 (Add) |  | 576 × 1408 |
| 84 | added |  | |  | Vision_FFN_20 (Feed Forward) | 16M | 576 × 1408 |
| 85 | added |  | |  | Vision_LN_21 (Layer Norm) | 2.8K | 576 × 1408 |
| 86 | added |  | |  | Vision_Attn_21 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 87 | added |  | |  | Vision_Add_21 (Add) |  | 576 × 1408 |
| 88 | added |  | |  | Vision_FFN_21 (Feed Forward) | 16M | 576 × 1408 |
| 89 | added |  | |  | Vision_LN_22 (Layer Norm) | 2.8K | 576 × 1408 |
| 90 | added |  | |  | Vision_Attn_22 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 91 | added |  | |  | Vision_Add_22 (Add) |  | 576 × 1408 |
| 92 | added |  | |  | Vision_FFN_22 (Feed Forward) | 16M | 576 × 1408 |
| 93 | added |  | |  | Vision_LN_23 (Layer Norm) | 2.8K | 576 × 1408 |
| 94 | added |  | |  | Vision_Attn_23 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 95 | added |  | |  | Vision_Add_23 (Add) |  | 576 × 1408 |
| 96 | added |  | |  | Vision_FFN_23 (Feed Forward) | 16M | 576 × 1408 |
| 97 | added |  | |  | Vision_LN_24 (Layer Norm) | 2.8K | 576 × 1408 |
| 98 | added |  | |  | Vision_Attn_24 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 99 | added |  | |  | Vision_Add_24 (Add) |  | 576 × 1408 |
| 100 | added |  | |  | Vision_FFN_24 (Feed Forward) | 16M | 576 × 1408 |
| 101 | added |  | |  | Vision_LN_25 (Layer Norm) | 2.8K | 576 × 1408 |
| 102 | added |  | |  | Vision_Attn_25 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 103 | added |  | |  | Vision_Add_25 (Add) |  | 576 × 1408 |
| 104 | added |  | |  | Vision_FFN_25 (Feed Forward) | 16M | 576 × 1408 |
| 105 | added |  | |  | Vision_LN_26 (Layer Norm) | 2.8K | 576 × 1408 |
| 106 | added |  | |  | Vision_Attn_26 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 107 | added |  | |  | Vision_Add_26 (Add) |  | 576 × 1408 |
| 108 | added |  | |  | Vision_FFN_26 (Feed Forward) | 16M | 576 × 1408 |
| 109 | added |  | |  | Vision_LN_27 (Layer Norm) | 2.8K | 576 × 1408 |
| 110 | added |  | |  | Vision_Attn_27 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 111 | added |  | |  | Vision_Add_27 (Add) |  | 576 × 1408 |
| 112 | added |  | |  | Vision_FFN_27 (Feed Forward) | 16M | 576 × 1408 |
| 113 | added |  | |  | Vision_LN_28 (Layer Norm) | 2.8K | 576 × 1408 |
| 114 | added |  | |  | Vision_Attn_28 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 115 | added |  | |  | Vision_Add_28 (Add) |  | 576 × 1408 |
| 116 | added |  | |  | Vision_FFN_28 (Feed Forward) | 16M | 576 × 1408 |
| 117 | added |  | |  | Vision_LN_29 (Layer Norm) | 2.8K | 576 × 1408 |
| 118 | added |  | |  | Vision_Attn_29 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 119 | added |  | |  | Vision_Add_29 (Add) |  | 576 × 1408 |
| 120 | added |  | |  | Vision_FFN_29 (Feed Forward) | 16M | 576 × 1408 |
| 121 | added |  | |  | Vision_LN_30 (Layer Norm) | 2.8K | 576 × 1408 |
| 122 | added |  | |  | Vision_Attn_30 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 123 | added |  | |  | Vision_Add_30 (Add) |  | 576 × 1408 |
| 124 | added |  | |  | Vision_FFN_30 (Feed Forward) | 16M | 576 × 1408 |
| 125 | added |  | |  | Vision_LN_31 (Layer Norm) | 2.8K | 576 × 1408 |
| 126 | added |  | |  | Vision_Attn_31 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 127 | added |  | |  | Vision_Add_31 (Add) |  | 576 × 1408 |
| 128 | added |  | |  | Vision_FFN_31 (Feed Forward) | 16M | 576 × 1408 |
| 129 | added |  | |  | Vision_LN_32 (Layer Norm) | 2.8K | 576 × 1408 |
| 130 | added |  | |  | Vision_Attn_32 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 131 | added |  | |  | Vision_Add_32 (Add) |  | 576 × 1408 |
| 132 | added |  | |  | Vision_FFN_32 (Feed Forward) | 16M | 576 × 1408 |
| 133 | added |  | |  | Vision_LN_33 (Layer Norm) | 2.8K | 576 × 1408 |
| 134 | added |  | |  | Vision_Attn_33 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 135 | added |  | |  | Vision_Add_33 (Add) |  | 576 × 1408 |
| 136 | added |  | |  | Vision_FFN_33 (Feed Forward) | 16M | 576 × 1408 |
| 137 | added |  | |  | Vision_LN_34 (Layer Norm) | 2.8K | 576 × 1408 |
| 138 | added |  | |  | Vision_Attn_34 (Multi Head Attention) | 7.9M | 576 × 1408 |
| 139 | added |  | |  | Vision_Add_34 (Add) |  | 576 × 1408 |
| 140 | added |  | |  | Vision_FFN_34 (Feed Forward) | 16M | 576 × 1408 |
| 141 | added |  | |  | Vision projector (Projection) | 7.2M | 576 × 5120 |
| 142 | added |  | |  | Vision tokens (Reshape) |  | 1 × 576 × 5120 |
| 143 | added |  | |  | Multimodal fusion (concat tokens) (Concatenate) |  | 1 × 10486336 × 5120 |
| 144 | changed (normalizedShape) | RMSNorm_1_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_1_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 145 | changed (embedDim, numHeads) | Attention_1 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_1 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 146 | same | Add_1_attn (Add) |  | 1 × 40960 × 4096 | Add_1_attn (Add) |  | 1 × 10486336 × 5120 |
| 147 | changed (normalizedShape) | RMSNorm_1_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_1_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 148 | removed | FFN_1 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 149 | added |  | |  | MoE_1 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 150 | same | Add_1_ffn (Add) |  | 1 × 40960 × 4096 | Add_1_ffn (Add) |  | 1 × 10486336 × 5120 |
| 151 | changed (normalizedShape) | RMSNorm_2_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_2_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 152 | changed (embedDim, numHeads) | Attention_2 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_2 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 153 | same | Add_2_attn (Add) |  | 1 × 40960 × 4096 | Add_2_attn (Add) |  | 1 × 10486336 × 5120 |
| 154 | changed (normalizedShape) | RMSNorm_2_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_2_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 155 | removed | FFN_2 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 156 | added |  | |  | MoE_2 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 157 | same | Add_2_ffn (Add) |  | 1 × 40960 × 4096 | Add_2_ffn (Add) |  | 1 × 10486336 × 5120 |
| 158 | changed (normalizedShape) | RMSNorm_3_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_3_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 159 | changed (embedDim, numHeads) | Attention_3 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_3 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 160 | same | Add_3_attn (Add) |  | 1 × 40960 × 4096 | Add_3_attn (Add) |  | 1 × 10486336 × 5120 |
| 161 | changed (normalizedShape) | RMSNorm_3_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_3_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 162 | removed | FFN_3 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 163 | added |  | |  | MoE_3 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 164 | same | Add_3_ffn (Add) |  | 1 × 40960 × 4096 | Add_3_ffn (Add) |  | 1 × 10486336 × 5120 |
| 165 | changed (normalizedShape) | RMSNorm_4_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_4_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 166 | changed (embedDim, numHeads) | Attention_4 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_4 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 167 | same | Add_4_attn (Add) |  | 1 × 40960 × 4096 | Add_4_attn (Add) |  | 1 × 10486336 × 5120 |
| 168 | changed (normalizedShape) | RMSNorm_4_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_4_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 169 | removed | FFN_4 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 170 | added |  | |  | MoE_4 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 171 | same | Add_4_ffn (Add) |  | 1 × 40960 × 4096 | Add_4_ffn (Add) |  | 1 × 10486336 × 5120 |
| 172 | changed (normalizedShape) | RMSNorm_5_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_5_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 173 | changed (embedDim, numHeads) | Attention_5 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_5 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 174 | same | Add_5_attn (Add) |  | 1 × 40960 × 4096 | Add_5_attn (Add) |  | 1 × 10486336 × 5120 |
| 175 | changed (normalizedShape) | RMSNorm_5_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_5_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 176 | removed | FFN_5 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 177 | added |  | |  | MoE_5 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 178 | same | Add_5_ffn (Add) |  | 1 × 40960 × 4096 | Add_5_ffn (Add) |  | 1 × 10486336 × 5120 |
| 179 | changed (normalizedShape) | RMSNorm_6_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_6_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 180 | changed (embedDim, numHeads) | Attention_6 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_6 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 181 | same | Add_6_attn (Add) |  | 1 × 40960 × 4096 | Add_6_attn (Add) |  | 1 × 10486336 × 5120 |
| 182 | changed (normalizedShape) | RMSNorm_6_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_6_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 183 | removed | FFN_6 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 184 | added |  | |  | MoE_6 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 185 | same | Add_6_ffn (Add) |  | 1 × 40960 × 4096 | Add_6_ffn (Add) |  | 1 × 10486336 × 5120 |
| 186 | changed (normalizedShape) | RMSNorm_7_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_7_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 187 | changed (embedDim, numHeads) | Attention_7 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_7 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 188 | same | Add_7_attn (Add) |  | 1 × 40960 × 4096 | Add_7_attn (Add) |  | 1 × 10486336 × 5120 |
| 189 | changed (normalizedShape) | RMSNorm_7_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_7_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 190 | removed | FFN_7 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 191 | added |  | |  | MoE_7 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 192 | same | Add_7_ffn (Add) |  | 1 × 40960 × 4096 | Add_7_ffn (Add) |  | 1 × 10486336 × 5120 |
| 193 | changed (normalizedShape) | RMSNorm_8_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_8_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 194 | changed (embedDim, numHeads) | Attention_8 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_8 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 195 | same | Add_8_attn (Add) |  | 1 × 40960 × 4096 | Add_8_attn (Add) |  | 1 × 10486336 × 5120 |
| 196 | changed (normalizedShape) | RMSNorm_8_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_8_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 197 | removed | FFN_8 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 198 | added |  | |  | MoE_8 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 199 | same | Add_8_ffn (Add) |  | 1 × 40960 × 4096 | Add_8_ffn (Add) |  | 1 × 10486336 × 5120 |
| 200 | changed (normalizedShape) | RMSNorm_9_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_9_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 201 | changed (embedDim, numHeads) | Attention_9 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_9 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 202 | same | Add_9_attn (Add) |  | 1 × 40960 × 4096 | Add_9_attn (Add) |  | 1 × 10486336 × 5120 |
| 203 | changed (normalizedShape) | RMSNorm_9_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_9_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 204 | removed | FFN_9 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 205 | added |  | |  | MoE_9 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 206 | same | Add_9_ffn (Add) |  | 1 × 40960 × 4096 | Add_9_ffn (Add) |  | 1 × 10486336 × 5120 |
| 207 | changed (normalizedShape) | RMSNorm_10_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_10_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 208 | changed (embedDim, numHeads) | Attention_10 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_10 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 209 | same | Add_10_attn (Add) |  | 1 × 40960 × 4096 | Add_10_attn (Add) |  | 1 × 10486336 × 5120 |
| 210 | changed (normalizedShape) | RMSNorm_10_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_10_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 211 | removed | FFN_10 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 212 | added |  | |  | MoE_10 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 213 | same | Add_10_ffn (Add) |  | 1 × 40960 × 4096 | Add_10_ffn (Add) |  | 1 × 10486336 × 5120 |
| 214 | changed (normalizedShape) | RMSNorm_11_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_11_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 215 | changed (embedDim, numHeads) | Attention_11 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_11 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 216 | same | Add_11_attn (Add) |  | 1 × 40960 × 4096 | Add_11_attn (Add) |  | 1 × 10486336 × 5120 |
| 217 | changed (normalizedShape) | RMSNorm_11_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_11_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 218 | removed | FFN_11 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 219 | added |  | |  | MoE_11 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 220 | same | Add_11_ffn (Add) |  | 1 × 40960 × 4096 | Add_11_ffn (Add) |  | 1 × 10486336 × 5120 |
| 221 | changed (normalizedShape) | RMSNorm_12_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_12_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 222 | changed (embedDim, numHeads) | Attention_12 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_12 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 223 | same | Add_12_attn (Add) |  | 1 × 40960 × 4096 | Add_12_attn (Add) |  | 1 × 10486336 × 5120 |
| 224 | changed (normalizedShape) | RMSNorm_12_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_12_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 225 | removed | FFN_12 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 226 | added |  | |  | MoE_12 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 227 | same | Add_12_ffn (Add) |  | 1 × 40960 × 4096 | Add_12_ffn (Add) |  | 1 × 10486336 × 5120 |
| 228 | changed (normalizedShape) | RMSNorm_13_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_13_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 229 | changed (embedDim, numHeads) | Attention_13 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_13 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 230 | same | Add_13_attn (Add) |  | 1 × 40960 × 4096 | Add_13_attn (Add) |  | 1 × 10486336 × 5120 |
| 231 | changed (normalizedShape) | RMSNorm_13_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_13_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 232 | removed | FFN_13 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 233 | added |  | |  | MoE_13 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 234 | same | Add_13_ffn (Add) |  | 1 × 40960 × 4096 | Add_13_ffn (Add) |  | 1 × 10486336 × 5120 |
| 235 | changed (normalizedShape) | RMSNorm_14_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_14_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 236 | changed (embedDim, numHeads) | Attention_14 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_14 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 237 | same | Add_14_attn (Add) |  | 1 × 40960 × 4096 | Add_14_attn (Add) |  | 1 × 10486336 × 5120 |
| 238 | changed (normalizedShape) | RMSNorm_14_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_14_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 239 | removed | FFN_14 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 240 | added |  | |  | MoE_14 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 241 | same | Add_14_ffn (Add) |  | 1 × 40960 × 4096 | Add_14_ffn (Add) |  | 1 × 10486336 × 5120 |
| 242 | changed (normalizedShape) | RMSNorm_15_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_15_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 243 | changed (embedDim, numHeads) | Attention_15 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_15 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 244 | same | Add_15_attn (Add) |  | 1 × 40960 × 4096 | Add_15_attn (Add) |  | 1 × 10486336 × 5120 |
| 245 | changed (normalizedShape) | RMSNorm_15_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_15_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 246 | removed | FFN_15 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 247 | added |  | |  | MoE_15 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 248 | same | Add_15_ffn (Add) |  | 1 × 40960 × 4096 | Add_15_ffn (Add) |  | 1 × 10486336 × 5120 |
| 249 | changed (normalizedShape) | RMSNorm_16_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_16_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 250 | changed (embedDim, numHeads) | Attention_16 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_16 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 251 | same | Add_16_attn (Add) |  | 1 × 40960 × 4096 | Add_16_attn (Add) |  | 1 × 10486336 × 5120 |
| 252 | changed (normalizedShape) | RMSNorm_16_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_16_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 253 | removed | FFN_16 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 254 | added |  | |  | MoE_16 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 255 | same | Add_16_ffn (Add) |  | 1 × 40960 × 4096 | Add_16_ffn (Add) |  | 1 × 10486336 × 5120 |
| 256 | changed (normalizedShape) | RMSNorm_17_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_17_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 257 | changed (embedDim, numHeads) | Attention_17 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_17 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 258 | same | Add_17_attn (Add) |  | 1 × 40960 × 4096 | Add_17_attn (Add) |  | 1 × 10486336 × 5120 |
| 259 | changed (normalizedShape) | RMSNorm_17_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_17_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 260 | removed | FFN_17 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 261 | added |  | |  | MoE_17 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 262 | same | Add_17_ffn (Add) |  | 1 × 40960 × 4096 | Add_17_ffn (Add) |  | 1 × 10486336 × 5120 |
| 263 | changed (normalizedShape) | RMSNorm_18_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_18_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 264 | changed (embedDim, numHeads) | Attention_18 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_18 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 265 | same | Add_18_attn (Add) |  | 1 × 40960 × 4096 | Add_18_attn (Add) |  | 1 × 10486336 × 5120 |
| 266 | changed (normalizedShape) | RMSNorm_18_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_18_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 267 | removed | FFN_18 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 268 | added |  | |  | MoE_18 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 269 | same | Add_18_ffn (Add) |  | 1 × 40960 × 4096 | Add_18_ffn (Add) |  | 1 × 10486336 × 5120 |
| 270 | changed (normalizedShape) | RMSNorm_19_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_19_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 271 | changed (embedDim, numHeads) | Attention_19 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_19 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 272 | same | Add_19_attn (Add) |  | 1 × 40960 × 4096 | Add_19_attn (Add) |  | 1 × 10486336 × 5120 |
| 273 | changed (normalizedShape) | RMSNorm_19_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_19_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 274 | removed | FFN_19 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 275 | added |  | |  | MoE_19 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 276 | same | Add_19_ffn (Add) |  | 1 × 40960 × 4096 | Add_19_ffn (Add) |  | 1 × 10486336 × 5120 |
| 277 | changed (normalizedShape) | RMSNorm_20_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_20_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 278 | changed (embedDim, numHeads) | Attention_20 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_20 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 279 | same | Add_20_attn (Add) |  | 1 × 40960 × 4096 | Add_20_attn (Add) |  | 1 × 10486336 × 5120 |
| 280 | changed (normalizedShape) | RMSNorm_20_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_20_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 281 | removed | FFN_20 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 282 | added |  | |  | MoE_20 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 283 | same | Add_20_ffn (Add) |  | 1 × 40960 × 4096 | Add_20_ffn (Add) |  | 1 × 10486336 × 5120 |
| 284 | changed (normalizedShape) | RMSNorm_21_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_21_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 285 | changed (embedDim, numHeads) | Attention_21 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_21 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 286 | same | Add_21_attn (Add) |  | 1 × 40960 × 4096 | Add_21_attn (Add) |  | 1 × 10486336 × 5120 |
| 287 | changed (normalizedShape) | RMSNorm_21_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_21_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 288 | removed | FFN_21 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 289 | added |  | |  | MoE_21 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 290 | same | Add_21_ffn (Add) |  | 1 × 40960 × 4096 | Add_21_ffn (Add) |  | 1 × 10486336 × 5120 |
| 291 | changed (normalizedShape) | RMSNorm_22_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_22_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 292 | changed (embedDim, numHeads) | Attention_22 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_22 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 293 | same | Add_22_attn (Add) |  | 1 × 40960 × 4096 | Add_22_attn (Add) |  | 1 × 10486336 × 5120 |
| 294 | changed (normalizedShape) | RMSNorm_22_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_22_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 295 | removed | FFN_22 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 296 | added |  | |  | MoE_22 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 297 | same | Add_22_ffn (Add) |  | 1 × 40960 × 4096 | Add_22_ffn (Add) |  | 1 × 10486336 × 5120 |
| 298 | changed (normalizedShape) | RMSNorm_23_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_23_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 299 | changed (embedDim, numHeads) | Attention_23 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_23 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 300 | same | Add_23_attn (Add) |  | 1 × 40960 × 4096 | Add_23_attn (Add) |  | 1 × 10486336 × 5120 |
| 301 | changed (normalizedShape) | RMSNorm_23_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_23_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 302 | removed | FFN_23 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 303 | added |  | |  | MoE_23 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 304 | same | Add_23_ffn (Add) |  | 1 × 40960 × 4096 | Add_23_ffn (Add) |  | 1 × 10486336 × 5120 |
| 305 | changed (normalizedShape) | RMSNorm_24_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_24_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 306 | changed (embedDim, numHeads) | Attention_24 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_24 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 307 | same | Add_24_attn (Add) |  | 1 × 40960 × 4096 | Add_24_attn (Add) |  | 1 × 10486336 × 5120 |
| 308 | changed (normalizedShape) | RMSNorm_24_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_24_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 309 | removed | FFN_24 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 310 | added |  | |  | MoE_24 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 311 | same | Add_24_ffn (Add) |  | 1 × 40960 × 4096 | Add_24_ffn (Add) |  | 1 × 10486336 × 5120 |
| 312 | changed (normalizedShape) | RMSNorm_25_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_25_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 313 | changed (embedDim, numHeads) | Attention_25 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_25 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 314 | same | Add_25_attn (Add) |  | 1 × 40960 × 4096 | Add_25_attn (Add) |  | 1 × 10486336 × 5120 |
| 315 | changed (normalizedShape) | RMSNorm_25_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_25_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 316 | removed | FFN_25 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 317 | added |  | |  | MoE_25 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 318 | same | Add_25_ffn (Add) |  | 1 × 40960 × 4096 | Add_25_ffn (Add) |  | 1 × 10486336 × 5120 |
| 319 | changed (normalizedShape) | RMSNorm_26_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_26_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 320 | changed (embedDim, numHeads) | Attention_26 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_26 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 321 | same | Add_26_attn (Add) |  | 1 × 40960 × 4096 | Add_26_attn (Add) |  | 1 × 10486336 × 5120 |
| 322 | changed (normalizedShape) | RMSNorm_26_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_26_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 323 | removed | FFN_26 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 324 | added |  | |  | MoE_26 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 325 | same | Add_26_ffn (Add) |  | 1 × 40960 × 4096 | Add_26_ffn (Add) |  | 1 × 10486336 × 5120 |
| 326 | changed (normalizedShape) | RMSNorm_27_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_27_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 327 | changed (embedDim, numHeads) | Attention_27 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_27 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 328 | same | Add_27_attn (Add) |  | 1 × 40960 × 4096 | Add_27_attn (Add) |  | 1 × 10486336 × 5120 |
| 329 | changed (normalizedShape) | RMSNorm_27_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_27_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 330 | removed | FFN_27 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 331 | added |  | |  | MoE_27 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 332 | same | Add_27_ffn (Add) |  | 1 × 40960 × 4096 | Add_27_ffn (Add) |  | 1 × 10486336 × 5120 |
| 333 | changed (normalizedShape) | RMSNorm_28_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_28_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 334 | changed (embedDim, numHeads) | Attention_28 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_28 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 335 | same | Add_28_attn (Add) |  | 1 × 40960 × 4096 | Add_28_attn (Add) |  | 1 × 10486336 × 5120 |
| 336 | changed (normalizedShape) | RMSNorm_28_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_28_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 337 | removed | FFN_28 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 338 | added |  | |  | MoE_28 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 339 | same | Add_28_ffn (Add) |  | 1 × 40960 × 4096 | Add_28_ffn (Add) |  | 1 × 10486336 × 5120 |
| 340 | changed (normalizedShape) | RMSNorm_29_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_29_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 341 | changed (embedDim, numHeads) | Attention_29 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_29 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 342 | same | Add_29_attn (Add) |  | 1 × 40960 × 4096 | Add_29_attn (Add) |  | 1 × 10486336 × 5120 |
| 343 | changed (normalizedShape) | RMSNorm_29_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_29_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 344 | removed | FFN_29 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 345 | added |  | |  | MoE_29 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 346 | same | Add_29_ffn (Add) |  | 1 × 40960 × 4096 | Add_29_ffn (Add) |  | 1 × 10486336 × 5120 |
| 347 | changed (normalizedShape) | RMSNorm_30_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_30_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 348 | changed (embedDim, numHeads) | Attention_30 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_30 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 349 | same | Add_30_attn (Add) |  | 1 × 40960 × 4096 | Add_30_attn (Add) |  | 1 × 10486336 × 5120 |
| 350 | changed (normalizedShape) | RMSNorm_30_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_30_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 351 | removed | FFN_30 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 352 | added |  | |  | MoE_30 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 353 | same | Add_30_ffn (Add) |  | 1 × 40960 × 4096 | Add_30_ffn (Add) |  | 1 × 10486336 × 5120 |
| 354 | changed (normalizedShape) | RMSNorm_31_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_31_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 355 | changed (embedDim, numHeads) | Attention_31 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_31 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 356 | same | Add_31_attn (Add) |  | 1 × 40960 × 4096 | Add_31_attn (Add) |  | 1 × 10486336 × 5120 |
| 357 | changed (normalizedShape) | RMSNorm_31_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_31_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 358 | removed | FFN_31 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 359 | added |  | |  | MoE_31 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 360 | same | Add_31_ffn (Add) |  | 1 × 40960 × 4096 | Add_31_ffn (Add) |  | 1 × 10486336 × 5120 |
| 361 | changed (normalizedShape) | RMSNorm_32_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_32_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 362 | changed (embedDim, numHeads) | Attention_32 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_32 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 363 | same | Add_32_attn (Add) |  | 1 × 40960 × 4096 | Add_32_attn (Add) |  | 1 × 10486336 × 5120 |
| 364 | changed (normalizedShape) | RMSNorm_32_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_32_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 365 | removed | FFN_32 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 366 | added |  | |  | MoE_32 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 367 | same | Add_32_ffn (Add) |  | 1 × 40960 × 4096 | Add_32_ffn (Add) |  | 1 × 10486336 × 5120 |
| 368 | changed (normalizedShape) | RMSNorm_33_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_33_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 369 | changed (embedDim, numHeads) | Attention_33 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_33 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 370 | same | Add_33_attn (Add) |  | 1 × 40960 × 4096 | Add_33_attn (Add) |  | 1 × 10486336 × 5120 |
| 371 | changed (normalizedShape) | RMSNorm_33_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_33_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 372 | removed | FFN_33 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 373 | added |  | |  | MoE_33 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 374 | same | Add_33_ffn (Add) |  | 1 × 40960 × 4096 | Add_33_ffn (Add) |  | 1 × 10486336 × 5120 |
| 375 | changed (normalizedShape) | RMSNorm_34_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_34_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 376 | changed (embedDim, numHeads) | Attention_34 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_34 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 377 | same | Add_34_attn (Add) |  | 1 × 40960 × 4096 | Add_34_attn (Add) |  | 1 × 10486336 × 5120 |
| 378 | changed (normalizedShape) | RMSNorm_34_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_34_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 379 | removed | FFN_34 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 380 | added |  | |  | MoE_34 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 381 | same | Add_34_ffn (Add) |  | 1 × 40960 × 4096 | Add_34_ffn (Add) |  | 1 × 10486336 × 5120 |
| 382 | changed (normalizedShape) | RMSNorm_35_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_35_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 383 | changed (embedDim, numHeads) | Attention_35 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_35 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 384 | same | Add_35_attn (Add) |  | 1 × 40960 × 4096 | Add_35_attn (Add) |  | 1 × 10486336 × 5120 |
| 385 | changed (normalizedShape) | RMSNorm_35_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_35_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 386 | removed | FFN_35 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 387 | added |  | |  | MoE_35 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 388 | same | Add_35_ffn (Add) |  | 1 × 40960 × 4096 | Add_35_ffn (Add) |  | 1 × 10486336 × 5120 |
| 389 | changed (normalizedShape) | RMSNorm_36_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_36_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 390 | changed (embedDim, numHeads) | Attention_36 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | Attention_36 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 391 | same | Add_36_attn (Add) |  | 1 × 40960 × 4096 | Add_36_attn (Add) |  | 1 × 10486336 × 5120 |
| 392 | changed (normalizedShape) | RMSNorm_36_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | RMSNorm_36_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 393 | removed | FFN_36 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 394 | added |  | |  | MoE_36 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 395 | added |  | |  | Add_36_ffn (Add) |  | 1 × 10486336 × 5120 |
| 396 | added |  | |  | RMSNorm_37_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 397 | added |  | |  | Attention_37 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 398 | added |  | |  | Add_37_attn (Add) |  | 1 × 10486336 × 5120 |
| 399 | added |  | |  | RMSNorm_37_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 400 | added |  | |  | MoE_37 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 401 | added |  | |  | Add_37_ffn (Add) |  | 1 × 10486336 × 5120 |
| 402 | added |  | |  | RMSNorm_38_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 403 | added |  | |  | Attention_38 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 404 | added |  | |  | Add_38_attn (Add) |  | 1 × 10486336 × 5120 |
| 405 | added |  | |  | RMSNorm_38_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 406 | added |  | |  | MoE_38 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 407 | added |  | |  | Add_38_ffn (Add) |  | 1 × 10486336 × 5120 |
| 408 | added |  | |  | RMSNorm_39_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 409 | added |  | |  | Attention_39 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 410 | added |  | |  | Add_39_attn (Add) |  | 1 × 10486336 × 5120 |
| 411 | added |  | |  | RMSNorm_39_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 412 | added |  | |  | MoE_39 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 413 | added |  | |  | Add_39_ffn (Add) |  | 1 × 10486336 × 5120 |
| 414 | added |  | |  | RMSNorm_40_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 415 | added |  | |  | Attention_40 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 416 | added |  | |  | Add_40_attn (Add) |  | 1 × 10486336 × 5120 |
| 417 | added |  | |  | RMSNorm_40_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 418 | added |  | |  | MoE_40 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 419 | added |  | |  | Add_40_ffn (Add) |  | 1 × 10486336 × 5120 |
| 420 | added |  | |  | RMSNorm_41_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 421 | added |  | |  | Attention_41 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 422 | added |  | |  | Add_41_attn (Add) |  | 1 × 10486336 × 5120 |
| 423 | added |  | |  | RMSNorm_41_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 424 | added |  | |  | MoE_41 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 425 | added |  | |  | Add_41_ffn (Add) |  | 1 × 10486336 × 5120 |
| 426 | added |  | |  | RMSNorm_42_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 427 | added |  | |  | Attention_42 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 428 | added |  | |  | Add_42_attn (Add) |  | 1 × 10486336 × 5120 |
| 429 | added |  | |  | RMSNorm_42_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 430 | added |  | |  | MoE_42 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 431 | added |  | |  | Add_42_ffn (Add) |  | 1 × 10486336 × 5120 |
| 432 | added |  | |  | RMSNorm_43_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 433 | added |  | |  | Attention_43 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 434 | added |  | |  | Add_43_attn (Add) |  | 1 × 10486336 × 5120 |
| 435 | added |  | |  | RMSNorm_43_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 436 | added |  | |  | MoE_43 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 437 | added |  | |  | Add_43_ffn (Add) |  | 1 × 10486336 × 5120 |
| 438 | added |  | |  | RMSNorm_44_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 439 | added |  | |  | Attention_44 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 440 | added |  | |  | Add_44_attn (Add) |  | 1 × 10486336 × 5120 |
| 441 | added |  | |  | RMSNorm_44_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 442 | added |  | |  | MoE_44 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 443 | added |  | |  | Add_44_ffn (Add) |  | 1 × 10486336 × 5120 |
| 444 | added |  | |  | RMSNorm_45_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 445 | added |  | |  | Attention_45 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 446 | added |  | |  | Add_45_attn (Add) |  | 1 × 10486336 × 5120 |
| 447 | added |  | |  | RMSNorm_45_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 448 | added |  | |  | MoE_45 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 449 | added |  | |  | Add_45_ffn (Add) |  | 1 × 10486336 × 5120 |
| 450 | added |  | |  | RMSNorm_46_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 451 | added |  | |  | Attention_46 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 452 | added |  | |  | Add_46_attn (Add) |  | 1 × 10486336 × 5120 |
| 453 | added |  | |  | RMSNorm_46_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 454 | added |  | |  | MoE_46 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 455 | added |  | |  | Add_46_ffn (Add) |  | 1 × 10486336 × 5120 |
| 456 | added |  | |  | RMSNorm_47_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 457 | added |  | |  | Attention_47 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 458 | added |  | |  | Add_47_attn (Add) |  | 1 × 10486336 × 5120 |
| 459 | added |  | |  | RMSNorm_47_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 460 | added |  | |  | MoE_47 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 461 | added |  | |  | Add_47_ffn (Add) |  | 1 × 10486336 × 5120 |
| 462 | added |  | |  | RMSNorm_48_1 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 463 | added |  | |  | Attention_48 (Grouped Query Attention) | 63M | 1 × 10486336 × 5120 |
| 464 | added |  | |  | Add_48_attn (Add) |  | 1 × 10486336 × 5120 |
| 465 | added |  | |  | RMSNorm_48_2 (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 466 | added |  | |  | MoE_48 (Moe Layer) | 2.0B | 1 × 10486336 × 5120 |
| 467 | same | Add_36_ffn (Add) |  | 1 × 40960 × 4096 | Add_48_ffn (Add) |  | 1 × 10486336 × 5120 |
| 468 | changed (normalizedShape) | Final_RMSNorm (Rms Norm) | 4.1K | 1 × 40960 × 4096 | Final_RMSNorm (Rms Norm) | 5.1K | 1 × 10486336 × 5120 |
| 469 | changed (inFeatures, outFeatures) | LM_Head (Linear) | 622M | 1 × 40960 × 151936 | LM_Head (Linear) | 1.0B | 1 × 10486336 × 202048 |
| 470 | same | Output (Output) |  | 1 × 40960 × 151936 | Output (Output) |  | 1 × 10486336 × 202048 |

## What this is not

- The two are priced at different declared inputs (1 × 40960 against 1 × 10485760), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

## Graphs

- Qwen3-8B: https://neurarch.com/templates/qwen3-8b/model.json
- Llama-4 Scout: https://neurarch.com/templates/llama-4-scout/model.json
- Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`