Comparisons / GPT-2 vs Qwen3-8B
GPT-2 vs Qwen3-8B
What seven years of scaling actually changed inside the decoder.
Qwen3-8B has 8.1B more parameters than GPT-2: 212 layers added, 3 removed, 6 changed.
Baseline
GPT-2
- Layers
- 10
- Parameters
- 84M
- Input
- 1 × 1024
- Output
- 1 × 1024 × 50257
- Forward-passes
- yes
- Est. train cost
- $1.82
Compared
Qwen3-8B
- Layers
- 219
- Parameters
- 8.2B
- Input
- 1 × 40960
- Output
- 1 × 40960 × 151936
- Forward-passes
- yes
- Est. train cost
- $20,863
The deltas
Every number is Qwen3-8B relative to GPT-2.
Parameters
+8.1B (97× the size)
Layers
+209
Added
212
Removed
3
Changed
6
Unchanged
3
Which GPUs each one fits
Each side is measured at its own declared input (1 × 1024 against 1 × 40960). Both columns are right about their own model; the difference between them is not a fact about the designs.
| GPU | GPT-2 | Qwen3-8B |
|---|---|---|
| T4 16GB | fits | no |
| A100 40GB | fits | no |
| H100 80GB | fits | no |
Layer by layer
Aligned in topological order. 3 of 224 rows are the same layer with the same parameters.
Show all 224 rows
| GPT-2 | Qwen3-8B | ||||||
|---|---|---|---|---|---|---|---|
| Layer | Params | Output | Layer | Params | Output | ||
| 1 | changed shape | tokens Input | 1 × 1024 | Input Input | 1 × 40960 | ||
| 2 | changed numEmbeddings, embeddingDim, vocabSize, maxSeqLen | token_embed Embedding | 39M | 1 × 1024 × 768 | Embedding Embedding | 622M | 1 × 40960 × 4096 |
| 3 | removed | pos_embed Positional Encoding | 1 × 1024 × 768 | — | |||
| 4 | changed type, normalizedShape | ln_1 Layer Norm | 1.5K | 1 × 1024 × 768 | RMSNorm_1_1 Rms Norm | 4.1K | 1 × 40960 × 4096 |
| 5 | removed | attn Causal Attention | 2.4M | 1 × 1024 × 768 | — | ||
| 6 | added | — | Attention_1 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 7 | same | residual_1 Add | 1 × 1024 × 768 | Add_1_attn Add | 1 × 40960 × 4096 | ||
| 8 | changed type, normalizedShape | ln_2 Layer Norm | 1.5K | 1 × 1024 × 768 | RMSNorm_1_2 Rms Norm | 4.1K | 1 × 40960 × 4096 |
| 9 | removed | mlp Feed Forward | 4.7M | 1 × 1024 × 768 | — | ||
| 10 | added | — | FFN_1 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 11 | added | — | Add_1_ffn Add | 1 × 40960 × 4096 | |||
| 12 | added | — | RMSNorm_2_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 13 | added | — | Attention_2 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 14 | added | — | Add_2_attn Add | 1 × 40960 × 4096 | |||
| 15 | added | — | RMSNorm_2_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 16 | added | — | FFN_2 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 17 | added | — | Add_2_ffn Add | 1 × 40960 × 4096 | |||
| 18 | added | — | RMSNorm_3_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 19 | added | — | Attention_3 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 20 | added | — | Add_3_attn Add | 1 × 40960 × 4096 | |||
| 21 | added | — | RMSNorm_3_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 22 | added | — | FFN_3 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 23 | added | — | Add_3_ffn Add | 1 × 40960 × 4096 | |||
| 24 | added | — | RMSNorm_4_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 25 | added | — | Attention_4 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 26 | added | — | Add_4_attn Add | 1 × 40960 × 4096 | |||
| 27 | added | — | RMSNorm_4_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 28 | added | — | FFN_4 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 29 | added | — | Add_4_ffn Add | 1 × 40960 × 4096 | |||
| 30 | added | — | RMSNorm_5_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 31 | added | — | Attention_5 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 32 | added | — | Add_5_attn Add | 1 × 40960 × 4096 | |||
| 33 | added | — | RMSNorm_5_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 34 | added | — | FFN_5 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 35 | added | — | Add_5_ffn Add | 1 × 40960 × 4096 | |||
| 36 | added | — | RMSNorm_6_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 37 | added | — | Attention_6 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 38 | added | — | Add_6_attn Add | 1 × 40960 × 4096 | |||
| 39 | added | — | RMSNorm_6_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 40 | added | — | FFN_6 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 41 | added | — | Add_6_ffn Add | 1 × 40960 × 4096 | |||
| 42 | added | — | RMSNorm_7_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 43 | added | — | Attention_7 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 44 | added | — | Add_7_attn Add | 1 × 40960 × 4096 | |||
| 45 | added | — | RMSNorm_7_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 46 | added | — | FFN_7 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 47 | added | — | Add_7_ffn Add | 1 × 40960 × 4096 | |||
| 48 | added | — | RMSNorm_8_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 49 | added | — | Attention_8 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 50 | added | — | Add_8_attn Add | 1 × 40960 × 4096 | |||
| 51 | added | — | RMSNorm_8_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 52 | added | — | FFN_8 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 53 | added | — | Add_8_ffn Add | 1 × 40960 × 4096 | |||
| 54 | added | — | RMSNorm_9_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 55 | added | — | Attention_9 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 56 | added | — | Add_9_attn Add | 1 × 40960 × 4096 | |||
| 57 | added | — | RMSNorm_9_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 58 | added | — | FFN_9 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 59 | added | — | Add_9_ffn Add | 1 × 40960 × 4096 | |||
| 60 | added | — | RMSNorm_10_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 61 | added | — | Attention_10 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 62 | added | — | Add_10_attn Add | 1 × 40960 × 4096 | |||
| 63 | added | — | RMSNorm_10_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 64 | added | — | FFN_10 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 65 | added | — | Add_10_ffn Add | 1 × 40960 × 4096 | |||
| 66 | added | — | RMSNorm_11_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 67 | added | — | Attention_11 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 68 | added | — | Add_11_attn Add | 1 × 40960 × 4096 | |||
| 69 | added | — | RMSNorm_11_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 70 | added | — | FFN_11 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 71 | added | — | Add_11_ffn Add | 1 × 40960 × 4096 | |||
| 72 | added | — | RMSNorm_12_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 73 | added | — | Attention_12 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 74 | added | — | Add_12_attn Add | 1 × 40960 × 4096 | |||
| 75 | added | — | RMSNorm_12_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 76 | added | — | FFN_12 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 77 | added | — | Add_12_ffn Add | 1 × 40960 × 4096 | |||
| 78 | added | — | RMSNorm_13_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 79 | added | — | Attention_13 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 80 | added | — | Add_13_attn Add | 1 × 40960 × 4096 | |||
| 81 | added | — | RMSNorm_13_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 82 | added | — | FFN_13 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 83 | added | — | Add_13_ffn Add | 1 × 40960 × 4096 | |||
| 84 | added | — | RMSNorm_14_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 85 | added | — | Attention_14 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 86 | added | — | Add_14_attn Add | 1 × 40960 × 4096 | |||
| 87 | added | — | RMSNorm_14_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 88 | added | — | FFN_14 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 89 | added | — | Add_14_ffn Add | 1 × 40960 × 4096 | |||
| 90 | added | — | RMSNorm_15_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 91 | added | — | Attention_15 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 92 | added | — | Add_15_attn Add | 1 × 40960 × 4096 | |||
| 93 | added | — | RMSNorm_15_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 94 | added | — | FFN_15 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 95 | added | — | Add_15_ffn Add | 1 × 40960 × 4096 | |||
| 96 | added | — | RMSNorm_16_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 97 | added | — | Attention_16 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 98 | added | — | Add_16_attn Add | 1 × 40960 × 4096 | |||
| 99 | added | — | RMSNorm_16_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 100 | added | — | FFN_16 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 101 | added | — | Add_16_ffn Add | 1 × 40960 × 4096 | |||
| 102 | added | — | RMSNorm_17_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 103 | added | — | Attention_17 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 104 | added | — | Add_17_attn Add | 1 × 40960 × 4096 | |||
| 105 | added | — | RMSNorm_17_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 106 | added | — | FFN_17 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 107 | added | — | Add_17_ffn Add | 1 × 40960 × 4096 | |||
| 108 | added | — | RMSNorm_18_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 109 | added | — | Attention_18 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 110 | added | — | Add_18_attn Add | 1 × 40960 × 4096 | |||
| 111 | added | — | RMSNorm_18_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 112 | added | — | FFN_18 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 113 | added | — | Add_18_ffn Add | 1 × 40960 × 4096 | |||
| 114 | added | — | RMSNorm_19_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 115 | added | — | Attention_19 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 116 | added | — | Add_19_attn Add | 1 × 40960 × 4096 | |||
| 117 | added | — | RMSNorm_19_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 118 | added | — | FFN_19 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 119 | added | — | Add_19_ffn Add | 1 × 40960 × 4096 | |||
| 120 | added | — | RMSNorm_20_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 121 | added | — | Attention_20 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 122 | added | — | Add_20_attn Add | 1 × 40960 × 4096 | |||
| 123 | added | — | RMSNorm_20_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 124 | added | — | FFN_20 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 125 | added | — | Add_20_ffn Add | 1 × 40960 × 4096 | |||
| 126 | added | — | RMSNorm_21_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 127 | added | — | Attention_21 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 128 | added | — | Add_21_attn Add | 1 × 40960 × 4096 | |||
| 129 | added | — | RMSNorm_21_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 130 | added | — | FFN_21 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 131 | added | — | Add_21_ffn Add | 1 × 40960 × 4096 | |||
| 132 | added | — | RMSNorm_22_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 133 | added | — | Attention_22 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 134 | added | — | Add_22_attn Add | 1 × 40960 × 4096 | |||
| 135 | added | — | RMSNorm_22_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 136 | added | — | FFN_22 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 137 | added | — | Add_22_ffn Add | 1 × 40960 × 4096 | |||
| 138 | added | — | RMSNorm_23_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 139 | added | — | Attention_23 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 140 | added | — | Add_23_attn Add | 1 × 40960 × 4096 | |||
| 141 | added | — | RMSNorm_23_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 142 | added | — | FFN_23 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 143 | added | — | Add_23_ffn Add | 1 × 40960 × 4096 | |||
| 144 | added | — | RMSNorm_24_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 145 | added | — | Attention_24 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 146 | added | — | Add_24_attn Add | 1 × 40960 × 4096 | |||
| 147 | added | — | RMSNorm_24_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 148 | added | — | FFN_24 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 149 | added | — | Add_24_ffn Add | 1 × 40960 × 4096 | |||
| 150 | added | — | RMSNorm_25_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 151 | added | — | Attention_25 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 152 | added | — | Add_25_attn Add | 1 × 40960 × 4096 | |||
| 153 | added | — | RMSNorm_25_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 154 | added | — | FFN_25 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 155 | added | — | Add_25_ffn Add | 1 × 40960 × 4096 | |||
| 156 | added | — | RMSNorm_26_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 157 | added | — | Attention_26 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 158 | added | — | Add_26_attn Add | 1 × 40960 × 4096 | |||
| 159 | added | — | RMSNorm_26_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 160 | added | — | FFN_26 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 161 | added | — | Add_26_ffn Add | 1 × 40960 × 4096 | |||
| 162 | added | — | RMSNorm_27_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 163 | added | — | Attention_27 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 164 | added | — | Add_27_attn Add | 1 × 40960 × 4096 | |||
| 165 | added | — | RMSNorm_27_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 166 | added | — | FFN_27 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 167 | added | — | Add_27_ffn Add | 1 × 40960 × 4096 | |||
| 168 | added | — | RMSNorm_28_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 169 | added | — | Attention_28 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 170 | added | — | Add_28_attn Add | 1 × 40960 × 4096 | |||
| 171 | added | — | RMSNorm_28_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 172 | added | — | FFN_28 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 173 | added | — | Add_28_ffn Add | 1 × 40960 × 4096 | |||
| 174 | added | — | RMSNorm_29_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 175 | added | — | Attention_29 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 176 | added | — | Add_29_attn Add | 1 × 40960 × 4096 | |||
| 177 | added | — | RMSNorm_29_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 178 | added | — | FFN_29 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 179 | added | — | Add_29_ffn Add | 1 × 40960 × 4096 | |||
| 180 | added | — | RMSNorm_30_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 181 | added | — | Attention_30 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 182 | added | — | Add_30_attn Add | 1 × 40960 × 4096 | |||
| 183 | added | — | RMSNorm_30_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 184 | added | — | FFN_30 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 185 | added | — | Add_30_ffn Add | 1 × 40960 × 4096 | |||
| 186 | added | — | RMSNorm_31_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 187 | added | — | Attention_31 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 188 | added | — | Add_31_attn Add | 1 × 40960 × 4096 | |||
| 189 | added | — | RMSNorm_31_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 190 | added | — | FFN_31 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 191 | added | — | Add_31_ffn Add | 1 × 40960 × 4096 | |||
| 192 | added | — | RMSNorm_32_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 193 | added | — | Attention_32 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 194 | added | — | Add_32_attn Add | 1 × 40960 × 4096 | |||
| 195 | added | — | RMSNorm_32_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 196 | added | — | FFN_32 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 197 | added | — | Add_32_ffn Add | 1 × 40960 × 4096 | |||
| 198 | added | — | RMSNorm_33_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 199 | added | — | Attention_33 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 200 | added | — | Add_33_attn Add | 1 × 40960 × 4096 | |||
| 201 | added | — | RMSNorm_33_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 202 | added | — | FFN_33 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 203 | added | — | Add_33_ffn Add | 1 × 40960 × 4096 | |||
| 204 | added | — | RMSNorm_34_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 205 | added | — | Attention_34 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 206 | added | — | Add_34_attn Add | 1 × 40960 × 4096 | |||
| 207 | added | — | RMSNorm_34_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 208 | added | — | FFN_34 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 209 | added | — | Add_34_ffn Add | 1 × 40960 × 4096 | |||
| 210 | added | — | RMSNorm_35_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 211 | added | — | Attention_35 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 212 | added | — | Add_35_attn Add | 1 × 40960 × 4096 | |||
| 213 | added | — | RMSNorm_35_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 214 | added | — | FFN_35 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 215 | added | — | Add_35_ffn Add | 1 × 40960 × 4096 | |||
| 216 | added | — | RMSNorm_36_1 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 217 | added | — | Attention_36 Grouped Query Attention | 42M | 1 × 40960 × 4096 | ||
| 218 | added | — | Add_36_attn Add | 1 × 40960 × 4096 | |||
| 219 | added | — | RMSNorm_36_2 Rms Norm | 4.1K | 1 × 40960 × 4096 | ||
| 220 | added | — | FFN_36 Swiglu | 151M | 1 × 40960 × 4096 | ||
| 221 | same | residual_2 Add | 1 × 1024 × 768 | Add_36_ffn Add | 1 × 40960 × 4096 | ||
| 222 | changed type, normalizedShape | ln_f Layer Norm | 1.5K | 1 × 1024 × 768 | Final_RMSNorm Rms Norm | 4.1K | 1 × 40960 × 4096 |
| 223 | changed outFeatures, inFeatures, bias | lm_head Linear | 1 × 1024 × 50257 | LM_Head Linear | 622M | 1 × 40960 × 151936 | |
| 224 | same | logits Output | 1 × 1024 × 50257 | Output Output | 1 × 40960 × 151936 | ||
What this is not
- The two are priced at different declared inputs (1 × 1024 against 1 × 40960), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.
Take it further
Open either graph in the editor, change it, and check it again: GPT-2 · Qwen3-8B
Compare any two models of your own, including anything on Hugging Face: the comparison tool.
Machine-readable: this page as markdown ·
the pair index ·
POST https://www.neurarch.com/api/v1/plan for a graph of your own.