Comparisons / Qwen3-8B vs Phi-3 Mini Block
Qwen3-8B vs Phi-3 Mini Block
What gets cut to make a small model small.
Phi-3 Mini Block has 7.9B fewer parameters than Qwen3-8B: 1 layer added, 210 removed, 8 changed.
Baseline
Qwen3-8B
- Layers
- 219
- Parameters
- 8.2B
- Input
- 1 × 40960
- Output
- 1 × 40960 × 151936
- Forward-passes
- yes
- Est. train cost
- $20,863
Compared
Phi-3 Mini Block
- Layers
- 10
- Parameters
- 310M
- Input
- 1 × 2048
- Output
- 1 × 2048 × 32064
- Forward-passes
- yes
- Est. train cost
- $16.76
The deltas
Every number is Phi-3 Mini Block relative to Qwen3-8B.
Parameters
-7.9B (-96.2%)
Layers
-209
Added
1
Removed
210
Changed
8
Unchanged
3
Which GPUs each one fits
Each side is measured at its own declared input (1 × 40960 against 1 × 2048). Both columns are right about their own model; the difference between them is not a fact about the designs.
| GPU | Qwen3-8B | Phi-3 Mini Block |
|---|---|---|
| T4 16GB | no | no |
| A100 40GB | no | fits |
| H100 80GB | no | fits |
Layer by layer
Aligned in topological order. 3 of 222 rows are the same layer with the same parameters.
Show all 222 rows
| Qwen3-8B | Phi-3 Mini Block | ||||||
|---|---|---|---|---|---|---|---|
| Layer | Params | Output | Layer | Params | Output | ||
| 1 | changed shape | Input Input | 1 × 40960 | tokens Input | 1 × 2048 | ||
| 2 | added | — | rope Rope | ||||
| 3 | changed vocabSize, embeddingDim, maxSeqLen, numEmbeddings | Embedding Embedding | 622M | 1 × 40960 × 4096 | embed Embedding | 99M | 1 × 2048 × 3072 |
| 4 | removed | RMSNorm_1_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 5 | removed | Attention_1 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 6 | removed | Add_1_attn Add | 1 × 40960 × 4096 | — | |||
| 7 | removed | RMSNorm_1_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 8 | removed | FFN_1 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 9 | removed | Add_1_ffn Add | 1 × 40960 × 4096 | — | |||
| 10 | removed | RMSNorm_2_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 11 | removed | Attention_2 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 12 | removed | Add_2_attn Add | 1 × 40960 × 4096 | — | |||
| 13 | removed | RMSNorm_2_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 14 | removed | FFN_2 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 15 | removed | Add_2_ffn Add | 1 × 40960 × 4096 | — | |||
| 16 | removed | RMSNorm_3_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 17 | removed | Attention_3 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 18 | removed | Add_3_attn Add | 1 × 40960 × 4096 | — | |||
| 19 | removed | RMSNorm_3_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 20 | removed | FFN_3 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 21 | removed | Add_3_ffn Add | 1 × 40960 × 4096 | — | |||
| 22 | removed | RMSNorm_4_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 23 | removed | Attention_4 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 24 | removed | Add_4_attn Add | 1 × 40960 × 4096 | — | |||
| 25 | removed | RMSNorm_4_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 26 | removed | FFN_4 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 27 | removed | Add_4_ffn Add | 1 × 40960 × 4096 | — | |||
| 28 | removed | RMSNorm_5_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 29 | removed | Attention_5 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 30 | removed | Add_5_attn Add | 1 × 40960 × 4096 | — | |||
| 31 | removed | RMSNorm_5_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 32 | removed | FFN_5 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 33 | removed | Add_5_ffn Add | 1 × 40960 × 4096 | — | |||
| 34 | removed | RMSNorm_6_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 35 | removed | Attention_6 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 36 | removed | Add_6_attn Add | 1 × 40960 × 4096 | — | |||
| 37 | removed | RMSNorm_6_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 38 | removed | FFN_6 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 39 | removed | Add_6_ffn Add | 1 × 40960 × 4096 | — | |||
| 40 | removed | RMSNorm_7_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 41 | removed | Attention_7 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 42 | removed | Add_7_attn Add | 1 × 40960 × 4096 | — | |||
| 43 | removed | RMSNorm_7_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 44 | removed | FFN_7 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 45 | removed | Add_7_ffn Add | 1 × 40960 × 4096 | — | |||
| 46 | removed | RMSNorm_8_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 47 | removed | Attention_8 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 48 | removed | Add_8_attn Add | 1 × 40960 × 4096 | — | |||
| 49 | removed | RMSNorm_8_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 50 | removed | FFN_8 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 51 | removed | Add_8_ffn Add | 1 × 40960 × 4096 | — | |||
| 52 | removed | RMSNorm_9_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 53 | removed | Attention_9 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 54 | removed | Add_9_attn Add | 1 × 40960 × 4096 | — | |||
| 55 | removed | RMSNorm_9_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 56 | removed | FFN_9 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 57 | removed | Add_9_ffn Add | 1 × 40960 × 4096 | — | |||
| 58 | removed | RMSNorm_10_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 59 | removed | Attention_10 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 60 | removed | Add_10_attn Add | 1 × 40960 × 4096 | — | |||
| 61 | removed | RMSNorm_10_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 62 | removed | FFN_10 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 63 | removed | Add_10_ffn Add | 1 × 40960 × 4096 | — | |||
| 64 | removed | RMSNorm_11_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 65 | removed | Attention_11 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 66 | removed | Add_11_attn Add | 1 × 40960 × 4096 | — | |||
| 67 | removed | RMSNorm_11_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 68 | removed | FFN_11 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 69 | removed | Add_11_ffn Add | 1 × 40960 × 4096 | — | |||
| 70 | removed | RMSNorm_12_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 71 | removed | Attention_12 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 72 | removed | Add_12_attn Add | 1 × 40960 × 4096 | — | |||
| 73 | removed | RMSNorm_12_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 74 | removed | FFN_12 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 75 | removed | Add_12_ffn Add | 1 × 40960 × 4096 | — | |||
| 76 | removed | RMSNorm_13_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 77 | removed | Attention_13 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 78 | removed | Add_13_attn Add | 1 × 40960 × 4096 | — | |||
| 79 | removed | RMSNorm_13_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 80 | removed | FFN_13 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 81 | removed | Add_13_ffn Add | 1 × 40960 × 4096 | — | |||
| 82 | removed | RMSNorm_14_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 83 | removed | Attention_14 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 84 | removed | Add_14_attn Add | 1 × 40960 × 4096 | — | |||
| 85 | removed | RMSNorm_14_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 86 | removed | FFN_14 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 87 | removed | Add_14_ffn Add | 1 × 40960 × 4096 | — | |||
| 88 | removed | RMSNorm_15_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 89 | removed | Attention_15 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 90 | removed | Add_15_attn Add | 1 × 40960 × 4096 | — | |||
| 91 | removed | RMSNorm_15_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 92 | removed | FFN_15 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 93 | removed | Add_15_ffn Add | 1 × 40960 × 4096 | — | |||
| 94 | removed | RMSNorm_16_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 95 | removed | Attention_16 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 96 | removed | Add_16_attn Add | 1 × 40960 × 4096 | — | |||
| 97 | removed | RMSNorm_16_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 98 | removed | FFN_16 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 99 | removed | Add_16_ffn Add | 1 × 40960 × 4096 | — | |||
| 100 | removed | RMSNorm_17_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 101 | removed | Attention_17 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 102 | removed | Add_17_attn Add | 1 × 40960 × 4096 | — | |||
| 103 | removed | RMSNorm_17_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 104 | removed | FFN_17 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 105 | removed | Add_17_ffn Add | 1 × 40960 × 4096 | — | |||
| 106 | removed | RMSNorm_18_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 107 | removed | Attention_18 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 108 | removed | Add_18_attn Add | 1 × 40960 × 4096 | — | |||
| 109 | removed | RMSNorm_18_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 110 | removed | FFN_18 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 111 | removed | Add_18_ffn Add | 1 × 40960 × 4096 | — | |||
| 112 | removed | RMSNorm_19_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 113 | removed | Attention_19 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 114 | removed | Add_19_attn Add | 1 × 40960 × 4096 | — | |||
| 115 | removed | RMSNorm_19_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 116 | removed | FFN_19 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 117 | removed | Add_19_ffn Add | 1 × 40960 × 4096 | — | |||
| 118 | removed | RMSNorm_20_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 119 | removed | Attention_20 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 120 | removed | Add_20_attn Add | 1 × 40960 × 4096 | — | |||
| 121 | removed | RMSNorm_20_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 122 | removed | FFN_20 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 123 | removed | Add_20_ffn Add | 1 × 40960 × 4096 | — | |||
| 124 | removed | RMSNorm_21_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 125 | removed | Attention_21 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 126 | removed | Add_21_attn Add | 1 × 40960 × 4096 | — | |||
| 127 | removed | RMSNorm_21_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 128 | removed | FFN_21 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 129 | removed | Add_21_ffn Add | 1 × 40960 × 4096 | — | |||
| 130 | removed | RMSNorm_22_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 131 | removed | Attention_22 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 132 | removed | Add_22_attn Add | 1 × 40960 × 4096 | — | |||
| 133 | removed | RMSNorm_22_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 134 | removed | FFN_22 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 135 | removed | Add_22_ffn Add | 1 × 40960 × 4096 | — | |||
| 136 | removed | RMSNorm_23_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 137 | removed | Attention_23 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 138 | removed | Add_23_attn Add | 1 × 40960 × 4096 | — | |||
| 139 | removed | RMSNorm_23_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 140 | removed | FFN_23 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 141 | removed | Add_23_ffn Add | 1 × 40960 × 4096 | — | |||
| 142 | removed | RMSNorm_24_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 143 | removed | Attention_24 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 144 | removed | Add_24_attn Add | 1 × 40960 × 4096 | — | |||
| 145 | removed | RMSNorm_24_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 146 | removed | FFN_24 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 147 | removed | Add_24_ffn Add | 1 × 40960 × 4096 | — | |||
| 148 | removed | RMSNorm_25_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 149 | removed | Attention_25 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 150 | removed | Add_25_attn Add | 1 × 40960 × 4096 | — | |||
| 151 | removed | RMSNorm_25_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 152 | removed | FFN_25 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 153 | removed | Add_25_ffn Add | 1 × 40960 × 4096 | — | |||
| 154 | removed | RMSNorm_26_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 155 | removed | Attention_26 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 156 | removed | Add_26_attn Add | 1 × 40960 × 4096 | — | |||
| 157 | removed | RMSNorm_26_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 158 | removed | FFN_26 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 159 | removed | Add_26_ffn Add | 1 × 40960 × 4096 | — | |||
| 160 | removed | RMSNorm_27_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 161 | removed | Attention_27 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 162 | removed | Add_27_attn Add | 1 × 40960 × 4096 | — | |||
| 163 | removed | RMSNorm_27_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 164 | removed | FFN_27 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 165 | removed | Add_27_ffn Add | 1 × 40960 × 4096 | — | |||
| 166 | removed | RMSNorm_28_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 167 | removed | Attention_28 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 168 | removed | Add_28_attn Add | 1 × 40960 × 4096 | — | |||
| 169 | removed | RMSNorm_28_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 170 | removed | FFN_28 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 171 | removed | Add_28_ffn Add | 1 × 40960 × 4096 | — | |||
| 172 | removed | RMSNorm_29_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 173 | removed | Attention_29 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 174 | removed | Add_29_attn Add | 1 × 40960 × 4096 | — | |||
| 175 | removed | RMSNorm_29_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 176 | removed | FFN_29 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 177 | removed | Add_29_ffn Add | 1 × 40960 × 4096 | — | |||
| 178 | removed | RMSNorm_30_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 179 | removed | Attention_30 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 180 | removed | Add_30_attn Add | 1 × 40960 × 4096 | — | |||
| 181 | removed | RMSNorm_30_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 182 | removed | FFN_30 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 183 | removed | Add_30_ffn Add | 1 × 40960 × 4096 | — | |||
| 184 | removed | RMSNorm_31_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 185 | removed | Attention_31 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 186 | removed | Add_31_attn Add | 1 × 40960 × 4096 | — | |||
| 187 | removed | RMSNorm_31_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 188 | removed | FFN_31 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 189 | removed | Add_31_ffn Add | 1 × 40960 × 4096 | — | |||
| 190 | removed | RMSNorm_32_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 191 | removed | Attention_32 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 192 | removed | Add_32_attn Add | 1 × 40960 × 4096 | — | |||
| 193 | removed | RMSNorm_32_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 194 | removed | FFN_32 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 195 | removed | Add_32_ffn Add | 1 × 40960 × 4096 | — | |||
| 196 | removed | RMSNorm_33_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 197 | removed | Attention_33 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 198 | removed | Add_33_attn Add | 1 × 40960 × 4096 | — | |||
| 199 | removed | RMSNorm_33_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 200 | removed | FFN_33 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 201 | removed | Add_33_ffn Add | 1 × 40960 × 4096 | — | |||
| 202 | removed | RMSNorm_34_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 203 | removed | Attention_34 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 204 | removed | Add_34_attn Add | 1 × 40960 × 4096 | — | |||
| 205 | removed | RMSNorm_34_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 206 | removed | FFN_34 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 207 | removed | Add_34_ffn Add | 1 × 40960 × 4096 | — | |||
| 208 | removed | RMSNorm_35_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 209 | removed | Attention_35 Grouped Query Attention | 42M | 1 × 40960 × 4096 | — | ||
| 210 | removed | Add_35_attn Add | 1 × 40960 × 4096 | — | |||
| 211 | removed | RMSNorm_35_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | — | ||
| 212 | removed | FFN_35 Swiglu | 151M | 1 × 40960 × 4096 | — | ||
| 213 | removed | Add_35_ffn Add | 1 × 40960 × 4096 | — | |||
| 214 | changed normalizedShape | RMSNorm_36_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | norm_attn Rms Norm | 3.1K | 1 × 2048 × 3072 |
| 215 | changed embedDim, numKVHeads, headDim | Attention_36 Grouped Query Attention | 42M | 1 × 40960 × 4096 | attn Grouped Query Attention | 38M | 1 × 2048 × 3072 |
| 216 | same | Add_36_attn Add | 1 × 40960 × 4096 | residual_1 Add | 1 × 2048 × 3072 | ||
| 217 | changed normalizedShape | RMSNorm_36_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | norm_ffn Rms Norm | 3.1K | 1 × 2048 × 3072 |
| 218 | changed embedDim, dim, intermediateSize | FFN_36 Swiglu | 151M | 1 × 40960 × 4096 | ffn Swiglu | 75M | 1 × 2048 × 3072 |
| 219 | same | Add_36_ffn Add | 1 × 40960 × 4096 | residual_2 Add | 1 × 2048 × 3072 | ||
| 220 | changed normalizedShape | Final_RMSNorm Rms Norm | 4.1K | 1 × 40960 × 4096 | norm_out Rms Norm | 3.1K | 1 × 2048 × 3072 |
| 221 | changed inFeatures, outFeatures, bias | LM_Head Linear | 622M | 1 × 40960 × 151936 | lm_head Linear | 1 × 2048 × 32064 | |
| 222 | same | Output Output | 1 × 40960 × 151936 | output Output | 1 × 2048 × 32064 | ||
What this is not
- The two are priced at different declared inputs (1 × 40960 against 1 × 2048), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.
Take it further
Open either graph in the editor, change it, and check it again: Qwen3-8B · Phi-3 Mini Block
Compare any two models of your own, including anything on Hugging Face: the comparison tool.
Machine-readable: this page as markdown ·
the pair index ·
POST https://www.neurarch.com/api/v1/plan for a graph of your own.