# GPT-2 vs Qwen3-8B What seven years of scaling actually changed inside the decoder. **Qwen3-8B has 8.1B more parameters than GPT-2: 212 layers added, 3 removed, 6 changed.** Source: https://neurarch.com/diff/gpt2-vs-qwen3-8b.html ## Sides | | GPT-2 | Qwen3-8B | |---|---|---| | Layers | 10 | 219 | | Parameters | 84M | 8.2B | | Input | 1 × 1024 | 1 × 40960 | | Output | 1 × 1024 × 50257 | 1 × 40960 × 151936 | | Forward-passes | yes | yes | | Est. train cost | $1.82 | $20,863 | | T4 16GB | fits | no | | A100 40GB | fits | no | | H100 80GB | fits | no | ## Deltas (Qwen3-8B relative to GPT-2) - Parameters: +8.1B (97× the size) - Layers: +209 - Added 212, removed 3, changed 6, unchanged 3 ## Layer by layer | # | Status | GPT-2 | Params | Output | Qwen3-8B | Params | Output | |---|---|---|---|---|---|---|---| | 1 | changed (shape) | tokens (Input) | | 1 × 1024 | Input (Input) | | 1 × 40960 | | 2 | changed (numEmbeddings, embeddingDim, vocabSize, maxSeqLen) | token_embed (Embedding) | 39M | 1 × 1024 × 768 | Embedding (Embedding) | 622M | 1 × 40960 × 4096 | | 3 | removed | pos_embed (Positional Encoding) | | 1 × 1024 × 768 | | | | | 4 | changed (type, normalizedShape) | ln_1 (Layer Norm) | 1.5K | 1 × 1024 × 768 | RMSNorm_1_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 5 | removed | attn (Causal Attention) | 2.4M | 1 × 1024 × 768 | | | | | 6 | added | | | | Attention_1 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 7 | same | residual_1 (Add) | | 1 × 1024 × 768 | Add_1_attn (Add) | | 1 × 40960 × 4096 | | 8 | changed (type, normalizedShape) | ln_2 (Layer Norm) | 1.5K | 1 × 1024 × 768 | RMSNorm_1_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 9 | removed | mlp (Feed Forward) | 4.7M | 1 × 1024 × 768 | | | | | 10 | added | | | | FFN_1 (Swiglu) | 151M | 1 × 40960 × 4096 | | 11 | added | | | | Add_1_ffn (Add) | | 1 × 40960 × 4096 | | 12 | added | | | | RMSNorm_2_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 13 | added | | | | Attention_2 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 14 | added | | | | Add_2_attn (Add) | | 1 × 40960 × 4096 | | 15 | added | | | | RMSNorm_2_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 16 | added | | | | FFN_2 (Swiglu) | 151M | 1 × 40960 × 4096 | | 17 | added | | | | Add_2_ffn (Add) | | 1 × 40960 × 4096 | | 18 | added | | | | RMSNorm_3_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 19 | added | | | | Attention_3 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 20 | added | | | | Add_3_attn (Add) | | 1 × 40960 × 4096 | | 21 | added | | | | RMSNorm_3_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 22 | added | | | | FFN_3 (Swiglu) | 151M | 1 × 40960 × 4096 | | 23 | added | | | | Add_3_ffn (Add) | | 1 × 40960 × 4096 | | 24 | added | | | | RMSNorm_4_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 25 | added | | | | Attention_4 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 26 | added | | | | Add_4_attn (Add) | | 1 × 40960 × 4096 | | 27 | added | | | | RMSNorm_4_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 28 | added | | | | FFN_4 (Swiglu) | 151M | 1 × 40960 × 4096 | | 29 | added | | | | Add_4_ffn (Add) | | 1 × 40960 × 4096 | | 30 | added | | | | RMSNorm_5_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 31 | added | | | | Attention_5 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 32 | added | | | | Add_5_attn (Add) | | 1 × 40960 × 4096 | | 33 | added | | | | RMSNorm_5_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 34 | added | | | | FFN_5 (Swiglu) | 151M | 1 × 40960 × 4096 | | 35 | added | | | | Add_5_ffn (Add) | | 1 × 40960 × 4096 | | 36 | added | | | | RMSNorm_6_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 37 | added | | | | Attention_6 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 38 | added | | | | Add_6_attn (Add) | | 1 × 40960 × 4096 | | 39 | added | | | | RMSNorm_6_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 40 | added | | | | FFN_6 (Swiglu) | 151M | 1 × 40960 × 4096 | | 41 | added | | | | Add_6_ffn (Add) | | 1 × 40960 × 4096 | | 42 | added | | | | RMSNorm_7_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 43 | added | | | | Attention_7 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 44 | added | | | | Add_7_attn (Add) | | 1 × 40960 × 4096 | | 45 | added | | | | RMSNorm_7_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 46 | added | | | | FFN_7 (Swiglu) | 151M | 1 × 40960 × 4096 | | 47 | added | | | | Add_7_ffn (Add) | | 1 × 40960 × 4096 | | 48 | added | | | | RMSNorm_8_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 49 | added | | | | Attention_8 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 50 | added | | | | Add_8_attn (Add) | | 1 × 40960 × 4096 | | 51 | added | | | | RMSNorm_8_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 52 | added | | | | FFN_8 (Swiglu) | 151M | 1 × 40960 × 4096 | | 53 | added | | | | Add_8_ffn (Add) | | 1 × 40960 × 4096 | | 54 | added | | | | RMSNorm_9_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 55 | added | | | | Attention_9 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 56 | added | | | | Add_9_attn (Add) | | 1 × 40960 × 4096 | | 57 | added | | | | RMSNorm_9_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 58 | added | | | | FFN_9 (Swiglu) | 151M | 1 × 40960 × 4096 | | 59 | added | | | | Add_9_ffn (Add) | | 1 × 40960 × 4096 | | 60 | added | | | | RMSNorm_10_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 61 | added | | | | Attention_10 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 62 | added | | | | Add_10_attn (Add) | | 1 × 40960 × 4096 | | 63 | added | | | | RMSNorm_10_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 64 | added | | | | FFN_10 (Swiglu) | 151M | 1 × 40960 × 4096 | | 65 | added | | | | Add_10_ffn (Add) | | 1 × 40960 × 4096 | | 66 | added | | | | RMSNorm_11_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 67 | added | | | | Attention_11 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 68 | added | | | | Add_11_attn (Add) | | 1 × 40960 × 4096 | | 69 | added | | | | RMSNorm_11_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 70 | added | | | | FFN_11 (Swiglu) | 151M | 1 × 40960 × 4096 | | 71 | added | | | | Add_11_ffn (Add) | | 1 × 40960 × 4096 | | 72 | added | | | | RMSNorm_12_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 73 | added | | | | Attention_12 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 74 | added | | | | Add_12_attn (Add) | | 1 × 40960 × 4096 | | 75 | added | | | | RMSNorm_12_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 76 | added | | | | FFN_12 (Swiglu) | 151M | 1 × 40960 × 4096 | | 77 | added | | | | Add_12_ffn (Add) | | 1 × 40960 × 4096 | | 78 | added | | | | RMSNorm_13_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 79 | added | | | | Attention_13 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 80 | added | | | | Add_13_attn (Add) | | 1 × 40960 × 4096 | | 81 | added | | | | RMSNorm_13_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 82 | added | | | | FFN_13 (Swiglu) | 151M | 1 × 40960 × 4096 | | 83 | added | | | | Add_13_ffn (Add) | | 1 × 40960 × 4096 | | 84 | added | | | | RMSNorm_14_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 85 | added | | | | Attention_14 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 86 | added | | | | Add_14_attn (Add) | | 1 × 40960 × 4096 | | 87 | added | | | | RMSNorm_14_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 88 | added | | | | FFN_14 (Swiglu) | 151M | 1 × 40960 × 4096 | | 89 | added | | | | Add_14_ffn (Add) | | 1 × 40960 × 4096 | | 90 | added | | | | RMSNorm_15_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 91 | added | | | | Attention_15 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 92 | added | | | | Add_15_attn (Add) | | 1 × 40960 × 4096 | | 93 | added | | | | RMSNorm_15_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 94 | added | | | | FFN_15 (Swiglu) | 151M | 1 × 40960 × 4096 | | 95 | added | | | | Add_15_ffn (Add) | | 1 × 40960 × 4096 | | 96 | added | | | | RMSNorm_16_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 97 | added | | | | Attention_16 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 98 | added | | | | Add_16_attn (Add) | | 1 × 40960 × 4096 | | 99 | added | | | | RMSNorm_16_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 100 | added | | | | FFN_16 (Swiglu) | 151M | 1 × 40960 × 4096 | | 101 | added | | | | Add_16_ffn (Add) | | 1 × 40960 × 4096 | | 102 | added | | | | RMSNorm_17_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 103 | added | | | | Attention_17 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 104 | added | | | | Add_17_attn (Add) | | 1 × 40960 × 4096 | | 105 | added | | | | RMSNorm_17_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 106 | added | | | | FFN_17 (Swiglu) | 151M | 1 × 40960 × 4096 | | 107 | added | | | | Add_17_ffn (Add) | | 1 × 40960 × 4096 | | 108 | added | | | | RMSNorm_18_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 109 | added | | | | Attention_18 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 110 | added | | | | Add_18_attn (Add) | | 1 × 40960 × 4096 | | 111 | added | | | | RMSNorm_18_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 112 | added | | | | FFN_18 (Swiglu) | 151M | 1 × 40960 × 4096 | | 113 | added | | | | Add_18_ffn (Add) | | 1 × 40960 × 4096 | | 114 | added | | | | RMSNorm_19_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 115 | added | | | | Attention_19 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 116 | added | | | | Add_19_attn (Add) | | 1 × 40960 × 4096 | | 117 | added | | | | RMSNorm_19_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 118 | added | | | | FFN_19 (Swiglu) | 151M | 1 × 40960 × 4096 | | 119 | added | | | | Add_19_ffn (Add) | | 1 × 40960 × 4096 | | 120 | added | | | | RMSNorm_20_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 121 | added | | | | Attention_20 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 122 | added | | | | Add_20_attn (Add) | | 1 × 40960 × 4096 | | 123 | added | | | | RMSNorm_20_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 124 | added | | | | FFN_20 (Swiglu) | 151M | 1 × 40960 × 4096 | | 125 | added | | | | Add_20_ffn (Add) | | 1 × 40960 × 4096 | | 126 | added | | | | RMSNorm_21_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 127 | added | | | | Attention_21 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 128 | added | | | | Add_21_attn (Add) | | 1 × 40960 × 4096 | | 129 | added | | | | RMSNorm_21_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 130 | added | | | | FFN_21 (Swiglu) | 151M | 1 × 40960 × 4096 | | 131 | added | | | | Add_21_ffn (Add) | | 1 × 40960 × 4096 | | 132 | added | | | | RMSNorm_22_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 133 | added | | | | Attention_22 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 134 | added | | | | Add_22_attn (Add) | | 1 × 40960 × 4096 | | 135 | added | | | | RMSNorm_22_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 136 | added | | | | FFN_22 (Swiglu) | 151M | 1 × 40960 × 4096 | | 137 | added | | | | Add_22_ffn (Add) | | 1 × 40960 × 4096 | | 138 | added | | | | RMSNorm_23_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 139 | added | | | | Attention_23 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 140 | added | | | | Add_23_attn (Add) | | 1 × 40960 × 4096 | | 141 | added | | | | RMSNorm_23_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 142 | added | | | | FFN_23 (Swiglu) | 151M | 1 × 40960 × 4096 | | 143 | added | | | | Add_23_ffn (Add) | | 1 × 40960 × 4096 | | 144 | added | | | | RMSNorm_24_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 145 | added | | | | Attention_24 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 146 | added | | | | Add_24_attn (Add) | | 1 × 40960 × 4096 | | 147 | added | | | | RMSNorm_24_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 148 | added | | | | FFN_24 (Swiglu) | 151M | 1 × 40960 × 4096 | | 149 | added | | | | Add_24_ffn (Add) | | 1 × 40960 × 4096 | | 150 | added | | | | RMSNorm_25_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 151 | added | | | | Attention_25 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 152 | added | | | | Add_25_attn (Add) | | 1 × 40960 × 4096 | | 153 | added | | | | RMSNorm_25_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 154 | added | | | | FFN_25 (Swiglu) | 151M | 1 × 40960 × 4096 | | 155 | added | | | | Add_25_ffn (Add) | | 1 × 40960 × 4096 | | 156 | added | | | | RMSNorm_26_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 157 | added | | | | Attention_26 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 158 | added | | | | Add_26_attn (Add) | | 1 × 40960 × 4096 | | 159 | added | | | | RMSNorm_26_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 160 | added | | | | FFN_26 (Swiglu) | 151M | 1 × 40960 × 4096 | | 161 | added | | | | Add_26_ffn (Add) | | 1 × 40960 × 4096 | | 162 | added | | | | RMSNorm_27_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 163 | added | | | | Attention_27 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 164 | added | | | | Add_27_attn (Add) | | 1 × 40960 × 4096 | | 165 | added | | | | RMSNorm_27_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 166 | added | | | | FFN_27 (Swiglu) | 151M | 1 × 40960 × 4096 | | 167 | added | | | | Add_27_ffn (Add) | | 1 × 40960 × 4096 | | 168 | added | | | | RMSNorm_28_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 169 | added | | | | Attention_28 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 170 | added | | | | Add_28_attn (Add) | | 1 × 40960 × 4096 | | 171 | added | | | | RMSNorm_28_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 172 | added | | | | FFN_28 (Swiglu) | 151M | 1 × 40960 × 4096 | | 173 | added | | | | Add_28_ffn (Add) | | 1 × 40960 × 4096 | | 174 | added | | | | RMSNorm_29_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 175 | added | | | | Attention_29 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 176 | added | | | | Add_29_attn (Add) | | 1 × 40960 × 4096 | | 177 | added | | | | RMSNorm_29_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 178 | added | | | | FFN_29 (Swiglu) | 151M | 1 × 40960 × 4096 | | 179 | added | | | | Add_29_ffn (Add) | | 1 × 40960 × 4096 | | 180 | added | | | | RMSNorm_30_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 181 | added | | | | Attention_30 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 182 | added | | | | Add_30_attn (Add) | | 1 × 40960 × 4096 | | 183 | added | | | | RMSNorm_30_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 184 | added | | | | FFN_30 (Swiglu) | 151M | 1 × 40960 × 4096 | | 185 | added | | | | Add_30_ffn (Add) | | 1 × 40960 × 4096 | | 186 | added | | | | RMSNorm_31_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 187 | added | | | | Attention_31 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 188 | added | | | | Add_31_attn (Add) | | 1 × 40960 × 4096 | | 189 | added | | | | RMSNorm_31_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 190 | added | | | | FFN_31 (Swiglu) | 151M | 1 × 40960 × 4096 | | 191 | added | | | | Add_31_ffn (Add) | | 1 × 40960 × 4096 | | 192 | added | | | | RMSNorm_32_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 193 | added | | | | Attention_32 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 194 | added | | | | Add_32_attn (Add) | | 1 × 40960 × 4096 | | 195 | added | | | | RMSNorm_32_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 196 | added | | | | FFN_32 (Swiglu) | 151M | 1 × 40960 × 4096 | | 197 | added | | | | Add_32_ffn (Add) | | 1 × 40960 × 4096 | | 198 | added | | | | RMSNorm_33_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 199 | added | | | | Attention_33 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 200 | added | | | | Add_33_attn (Add) | | 1 × 40960 × 4096 | | 201 | added | | | | RMSNorm_33_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 202 | added | | | | FFN_33 (Swiglu) | 151M | 1 × 40960 × 4096 | | 203 | added | | | | Add_33_ffn (Add) | | 1 × 40960 × 4096 | | 204 | added | | | | RMSNorm_34_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 205 | added | | | | Attention_34 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 206 | added | | | | Add_34_attn (Add) | | 1 × 40960 × 4096 | | 207 | added | | | | RMSNorm_34_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 208 | added | | | | FFN_34 (Swiglu) | 151M | 1 × 40960 × 4096 | | 209 | added | | | | Add_34_ffn (Add) | | 1 × 40960 × 4096 | | 210 | added | | | | RMSNorm_35_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 211 | added | | | | Attention_35 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 212 | added | | | | Add_35_attn (Add) | | 1 × 40960 × 4096 | | 213 | added | | | | RMSNorm_35_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 214 | added | | | | FFN_35 (Swiglu) | 151M | 1 × 40960 × 4096 | | 215 | added | | | | Add_35_ffn (Add) | | 1 × 40960 × 4096 | | 216 | added | | | | RMSNorm_36_1 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 217 | added | | | | Attention_36 (Grouped Query Attention) | 42M | 1 × 40960 × 4096 | | 218 | added | | | | Add_36_attn (Add) | | 1 × 40960 × 4096 | | 219 | added | | | | RMSNorm_36_2 (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 220 | added | | | | FFN_36 (Swiglu) | 151M | 1 × 40960 × 4096 | | 221 | same | residual_2 (Add) | | 1 × 1024 × 768 | Add_36_ffn (Add) | | 1 × 40960 × 4096 | | 222 | changed (type, normalizedShape) | ln_f (Layer Norm) | 1.5K | 1 × 1024 × 768 | Final_RMSNorm (Rms Norm) | 4.1K | 1 × 40960 × 4096 | | 223 | changed (outFeatures, inFeatures, bias) | lm_head (Linear) | | 1 × 1024 × 50257 | LM_Head (Linear) | 622M | 1 × 40960 × 151936 | | 224 | same | logits (Output) | | 1 × 1024 × 50257 | Output (Output) | | 1 × 40960 × 151936 | ## What this is not - The two are priced at different declared inputs (1 × 1024 against 1 × 40960), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected. - Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not. - Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run. ## Graphs - GPT-2: https://neurarch.com/templates/gpt2/model.json - Qwen3-8B: https://neurarch.com/templates/qwen3-8b/model.json - Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`