# Qwen3-8B vs Phi-3 Mini Block

What gets cut to make a small model small.

**Phi-3 Mini Block has 7.9B fewer parameters than Qwen3-8B: 1 layer added, 210 removed, 8 changed.**

Source: https://neurarch.com/diff/qwen3-8b-vs-phi3-mini.html

## Sides

| | Qwen3-8B | Phi-3 Mini Block |
|---|---|---|
| Layers | 219 | 10 |
| Parameters | 8.2B | 310M |
| Input | 1 × 40960 | 1 × 2048 |
| Output | 1 × 40960 × 151936 | 1 × 2048 × 32064 |
| Forward-passes | yes | yes |
| Est. train cost | $20,863 | $16.76 |
| T4 16GB | no | fits |
| A100 40GB | no | fits |
| H100 80GB | no | fits |

## Deltas (Phi-3 Mini Block relative to Qwen3-8B)

- Parameters: -7.9B (-96.2%)
- Layers: -209
- Added 1, removed 210, changed 8, unchanged 3

## Layer by layer

| # | Status | Qwen3-8B | Params | Output | Phi-3 Mini Block | Params | Output |
|---|---|---|---|---|---|---|---|
| 1 | changed (shape) | Input (Input) |  | 1 × 40960 | tokens (Input) |  | 1 × 2048 |
| 2 | added |  | |  | rope (Rope) |  |  |
| 3 | changed (vocabSize, embeddingDim, maxSeqLen, numEmbeddings) | Embedding (Embedding) | 622M | 1 × 40960 × 4096 | embed (Embedding) | 99M | 1 × 2048 × 3072 |
| 4 | removed | RMSNorm_1_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 5 | removed | Attention_1 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 6 | removed | Add_1_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 7 | removed | RMSNorm_1_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 8 | removed | FFN_1 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 9 | removed | Add_1_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 10 | removed | RMSNorm_2_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 11 | removed | Attention_2 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 12 | removed | Add_2_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 13 | removed | RMSNorm_2_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 14 | removed | FFN_2 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 15 | removed | Add_2_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 16 | removed | RMSNorm_3_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 17 | removed | Attention_3 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 18 | removed | Add_3_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 19 | removed | RMSNorm_3_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 20 | removed | FFN_3 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 21 | removed | Add_3_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 22 | removed | RMSNorm_4_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 23 | removed | Attention_4 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 24 | removed | Add_4_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 25 | removed | RMSNorm_4_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 26 | removed | FFN_4 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 27 | removed | Add_4_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 28 | removed | RMSNorm_5_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 29 | removed | Attention_5 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 30 | removed | Add_5_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 31 | removed | RMSNorm_5_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 32 | removed | FFN_5 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 33 | removed | Add_5_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 34 | removed | RMSNorm_6_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 35 | removed | Attention_6 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 36 | removed | Add_6_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 37 | removed | RMSNorm_6_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 38 | removed | FFN_6 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 39 | removed | Add_6_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 40 | removed | RMSNorm_7_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 41 | removed | Attention_7 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 42 | removed | Add_7_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 43 | removed | RMSNorm_7_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 44 | removed | FFN_7 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 45 | removed | Add_7_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 46 | removed | RMSNorm_8_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 47 | removed | Attention_8 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 48 | removed | Add_8_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 49 | removed | RMSNorm_8_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 50 | removed | FFN_8 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 51 | removed | Add_8_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 52 | removed | RMSNorm_9_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 53 | removed | Attention_9 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 54 | removed | Add_9_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 55 | removed | RMSNorm_9_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 56 | removed | FFN_9 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 57 | removed | Add_9_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 58 | removed | RMSNorm_10_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 59 | removed | Attention_10 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 60 | removed | Add_10_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 61 | removed | RMSNorm_10_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 62 | removed | FFN_10 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 63 | removed | Add_10_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 64 | removed | RMSNorm_11_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 65 | removed | Attention_11 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 66 | removed | Add_11_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 67 | removed | RMSNorm_11_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 68 | removed | FFN_11 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 69 | removed | Add_11_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 70 | removed | RMSNorm_12_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 71 | removed | Attention_12 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 72 | removed | Add_12_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 73 | removed | RMSNorm_12_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 74 | removed | FFN_12 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 75 | removed | Add_12_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 76 | removed | RMSNorm_13_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 77 | removed | Attention_13 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 78 | removed | Add_13_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 79 | removed | RMSNorm_13_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 80 | removed | FFN_13 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 81 | removed | Add_13_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 82 | removed | RMSNorm_14_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 83 | removed | Attention_14 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 84 | removed | Add_14_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 85 | removed | RMSNorm_14_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 86 | removed | FFN_14 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 87 | removed | Add_14_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 88 | removed | RMSNorm_15_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 89 | removed | Attention_15 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 90 | removed | Add_15_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 91 | removed | RMSNorm_15_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 92 | removed | FFN_15 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 93 | removed | Add_15_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 94 | removed | RMSNorm_16_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 95 | removed | Attention_16 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 96 | removed | Add_16_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 97 | removed | RMSNorm_16_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 98 | removed | FFN_16 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 99 | removed | Add_16_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 100 | removed | RMSNorm_17_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 101 | removed | Attention_17 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 102 | removed | Add_17_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 103 | removed | RMSNorm_17_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 104 | removed | FFN_17 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 105 | removed | Add_17_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 106 | removed | RMSNorm_18_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 107 | removed | Attention_18 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 108 | removed | Add_18_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 109 | removed | RMSNorm_18_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 110 | removed | FFN_18 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 111 | removed | Add_18_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 112 | removed | RMSNorm_19_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 113 | removed | Attention_19 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 114 | removed | Add_19_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 115 | removed | RMSNorm_19_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 116 | removed | FFN_19 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 117 | removed | Add_19_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 118 | removed | RMSNorm_20_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 119 | removed | Attention_20 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 120 | removed | Add_20_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 121 | removed | RMSNorm_20_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 122 | removed | FFN_20 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 123 | removed | Add_20_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 124 | removed | RMSNorm_21_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 125 | removed | Attention_21 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 126 | removed | Add_21_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 127 | removed | RMSNorm_21_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 128 | removed | FFN_21 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 129 | removed | Add_21_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 130 | removed | RMSNorm_22_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 131 | removed | Attention_22 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 132 | removed | Add_22_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 133 | removed | RMSNorm_22_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 134 | removed | FFN_22 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 135 | removed | Add_22_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 136 | removed | RMSNorm_23_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 137 | removed | Attention_23 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 138 | removed | Add_23_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 139 | removed | RMSNorm_23_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 140 | removed | FFN_23 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 141 | removed | Add_23_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 142 | removed | RMSNorm_24_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 143 | removed | Attention_24 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 144 | removed | Add_24_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 145 | removed | RMSNorm_24_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 146 | removed | FFN_24 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 147 | removed | Add_24_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 148 | removed | RMSNorm_25_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 149 | removed | Attention_25 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 150 | removed | Add_25_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 151 | removed | RMSNorm_25_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 152 | removed | FFN_25 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 153 | removed | Add_25_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 154 | removed | RMSNorm_26_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 155 | removed | Attention_26 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 156 | removed | Add_26_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 157 | removed | RMSNorm_26_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 158 | removed | FFN_26 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 159 | removed | Add_26_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 160 | removed | RMSNorm_27_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 161 | removed | Attention_27 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 162 | removed | Add_27_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 163 | removed | RMSNorm_27_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 164 | removed | FFN_27 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 165 | removed | Add_27_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 166 | removed | RMSNorm_28_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 167 | removed | Attention_28 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 168 | removed | Add_28_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 169 | removed | RMSNorm_28_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 170 | removed | FFN_28 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 171 | removed | Add_28_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 172 | removed | RMSNorm_29_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 173 | removed | Attention_29 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 174 | removed | Add_29_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 175 | removed | RMSNorm_29_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 176 | removed | FFN_29 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 177 | removed | Add_29_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 178 | removed | RMSNorm_30_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 179 | removed | Attention_30 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 180 | removed | Add_30_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 181 | removed | RMSNorm_30_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 182 | removed | FFN_30 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 183 | removed | Add_30_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 184 | removed | RMSNorm_31_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 185 | removed | Attention_31 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 186 | removed | Add_31_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 187 | removed | RMSNorm_31_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 188 | removed | FFN_31 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 189 | removed | Add_31_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 190 | removed | RMSNorm_32_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 191 | removed | Attention_32 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 192 | removed | Add_32_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 193 | removed | RMSNorm_32_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 194 | removed | FFN_32 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 195 | removed | Add_32_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 196 | removed | RMSNorm_33_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 197 | removed | Attention_33 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 198 | removed | Add_33_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 199 | removed | RMSNorm_33_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 200 | removed | FFN_33 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 201 | removed | Add_33_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 202 | removed | RMSNorm_34_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 203 | removed | Attention_34 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 204 | removed | Add_34_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 205 | removed | RMSNorm_34_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 206 | removed | FFN_34 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 207 | removed | Add_34_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 208 | removed | RMSNorm_35_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 209 | removed | Attention_35 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 |  | |  |
| 210 | removed | Add_35_attn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 211 | removed | RMSNorm_35_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 |  | |  |
| 212 | removed | FFN_35 (Swiglu) | 151M | 1 × 40960 × 4096 |  | |  |
| 213 | removed | Add_35_ffn (Add) |  | 1 × 40960 × 4096 |  | |  |
| 214 | changed (normalizedShape) | RMSNorm_36_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | norm_attn (Rms Norm) | 3.1K | 1 × 2048 × 3072 |
| 215 | changed (embedDim, numKVHeads, headDim) | Attention_36 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | attn (Grouped Query Attention) | 38M | 1 × 2048 × 3072 |
| 216 | same | Add_36_attn (Add) |  | 1 × 40960 × 4096 | residual_1 (Add) |  | 1 × 2048 × 3072 |
| 217 | changed (normalizedShape) | RMSNorm_36_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | norm_ffn (Rms Norm) | 3.1K | 1 × 2048 × 3072 |
| 218 | changed (embedDim, dim, intermediateSize) | FFN_36 (Swiglu) | 151M | 1 × 40960 × 4096 | ffn (Swiglu) | 75M | 1 × 2048 × 3072 |
| 219 | same | Add_36_ffn (Add) |  | 1 × 40960 × 4096 | residual_2 (Add) |  | 1 × 2048 × 3072 |
| 220 | changed (normalizedShape) | Final_RMSNorm (Rms Norm) | 4.1K | 1 × 40960 × 4096 | norm_out (Rms Norm) | 3.1K | 1 × 2048 × 3072 |
| 221 | changed (inFeatures, outFeatures, bias) | LM_Head (Linear) | 622M | 1 × 40960 × 151936 | lm_head (Linear) |  | 1 × 2048 × 32064 |
| 222 | same | Output (Output) |  | 1 × 40960 × 151936 | output (Output) |  | 1 × 2048 × 32064 |

## What this is not

- The two are priced at different declared inputs (1 × 40960 against 1 × 2048), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

## Graphs

- Qwen3-8B: https://neurarch.com/templates/qwen3-8b/model.json
- Phi-3 Mini Block: https://neurarch.com/templates/phi3-mini/model.json
- Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`