# Qwen3-8B

> Modern dense decoder LLM — GQA with QK-RMSNorm on the query/key projections for training stability (Alibaba 2025)

A clean, current dense-decoder reference. Pick when you want a straightforward modern LLM block without MoE routing.

- Category: NLP/LLM
- Layers: 221
- Parameters: 8.19B
- Input shape (batchless): 1 × 40960
- Output shape: 1 × 40960 × 151936
- Verifier verdict: warn
- Graph JSON: https://neurarch.com/templates/qwen3-8b/model.json
- Open on the canvas: https://neurarch.com/?template=qwen3-8b

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | Input | Input | shape=[1, 40960] | 1 × 40960 |
| 2 | Embedding | Embedding | vocabSize=151936 | 1 × 40960 × 4096 |
| 3 | RMSNorm_1_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 4 | Attention_1 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 5 | Add_1_attn | Add |  | 1 × 40960 × 4096 |
| 6 | RMSNorm_1_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 7 | FFN_1 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 8 | Add_1_ffn | Add |  | 1 × 40960 × 4096 |
| 9 | RMSNorm_2_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 10 | Attention_2 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 11 | Add_2_attn | Add |  | 1 × 40960 × 4096 |
| 12 | RMSNorm_2_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 13 | FFN_2 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 14 | Add_2_ffn | Add |  | 1 × 40960 × 4096 |
| 15 | RMSNorm_3_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 16 | Attention_3 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 17 | Add_3_attn | Add |  | 1 × 40960 × 4096 |
| 18 | RMSNorm_3_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 19 | FFN_3 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 20 | Add_3_ffn | Add |  | 1 × 40960 × 4096 |
| 21 | RMSNorm_4_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 22 | Attention_4 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 23 | Add_4_attn | Add |  | 1 × 40960 × 4096 |
| 24 | RMSNorm_4_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 25 | FFN_4 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 26 | Add_4_ffn | Add |  | 1 × 40960 × 4096 |
| 27 | RMSNorm_5_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 28 | Attention_5 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 29 | Add_5_attn | Add |  | 1 × 40960 × 4096 |
| 30 | RMSNorm_5_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 31 | FFN_5 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 32 | Add_5_ffn | Add |  | 1 × 40960 × 4096 |
| 33 | RMSNorm_6_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 34 | Attention_6 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 35 | Add_6_attn | Add |  | 1 × 40960 × 4096 |
| 36 | RMSNorm_6_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 37 | FFN_6 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 38 | Add_6_ffn | Add |  | 1 × 40960 × 4096 |
| 39 | RMSNorm_7_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 40 | Attention_7 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 41 | Add_7_attn | Add |  | 1 × 40960 × 4096 |
| 42 | RMSNorm_7_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 43 | FFN_7 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 44 | Add_7_ffn | Add |  | 1 × 40960 × 4096 |
| 45 | RMSNorm_8_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 46 | Attention_8 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 47 | Add_8_attn | Add |  | 1 × 40960 × 4096 |
| 48 | RMSNorm_8_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 49 | FFN_8 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 50 | Add_8_ffn | Add |  | 1 × 40960 × 4096 |
| 51 | RMSNorm_9_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 52 | Attention_9 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 53 | Add_9_attn | Add |  | 1 × 40960 × 4096 |
| 54 | RMSNorm_9_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 55 | FFN_9 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 56 | Add_9_ffn | Add |  | 1 × 40960 × 4096 |
| 57 | RMSNorm_10_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 58 | Attention_10 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 59 | Add_10_attn | Add |  | 1 × 40960 × 4096 |
| 60 | RMSNorm_10_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 61 | FFN_10 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 62 | Add_10_ffn | Add |  | 1 × 40960 × 4096 |
| 63 | RMSNorm_11_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 64 | Attention_11 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 65 | Add_11_attn | Add |  | 1 × 40960 × 4096 |
| 66 | RMSNorm_11_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 67 | FFN_11 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 68 | Add_11_ffn | Add |  | 1 × 40960 × 4096 |
| 69 | RMSNorm_12_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 70 | Attention_12 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 71 | Add_12_attn | Add |  | 1 × 40960 × 4096 |
| 72 | RMSNorm_12_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 73 | FFN_12 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 74 | Add_12_ffn | Add |  | 1 × 40960 × 4096 |
| 75 | RMSNorm_13_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 76 | Attention_13 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 77 | Add_13_attn | Add |  | 1 × 40960 × 4096 |
| 78 | RMSNorm_13_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 79 | FFN_13 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 80 | Add_13_ffn | Add |  | 1 × 40960 × 4096 |
| 81 | RMSNorm_14_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 82 | Attention_14 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 83 | Add_14_attn | Add |  | 1 × 40960 × 4096 |
| 84 | RMSNorm_14_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 85 | FFN_14 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 86 | Add_14_ffn | Add |  | 1 × 40960 × 4096 |
| 87 | RMSNorm_15_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 88 | Attention_15 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 89 | Add_15_attn | Add |  | 1 × 40960 × 4096 |
| 90 | RMSNorm_15_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 91 | FFN_15 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 92 | Add_15_ffn | Add |  | 1 × 40960 × 4096 |
| 93 | RMSNorm_16_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 94 | Attention_16 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 95 | Add_16_attn | Add |  | 1 × 40960 × 4096 |
| 96 | RMSNorm_16_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 97 | FFN_16 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 98 | Add_16_ffn | Add |  | 1 × 40960 × 4096 |
| 99 | RMSNorm_17_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 100 | Attention_17 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 101 | Add_17_attn | Add |  | 1 × 40960 × 4096 |
| 102 | RMSNorm_17_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 103 | FFN_17 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 104 | Add_17_ffn | Add |  | 1 × 40960 × 4096 |
| 105 | RMSNorm_18_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 106 | Attention_18 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 107 | Add_18_attn | Add |  | 1 × 40960 × 4096 |
| 108 | RMSNorm_18_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 109 | FFN_18 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 110 | Add_18_ffn | Add |  | 1 × 40960 × 4096 |
| 111 | RMSNorm_19_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 112 | Attention_19 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 113 | Add_19_attn | Add |  | 1 × 40960 × 4096 |
| 114 | RMSNorm_19_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 115 | FFN_19 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 116 | Add_19_ffn | Add |  | 1 × 40960 × 4096 |
| 117 | RMSNorm_20_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 118 | Attention_20 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 119 | Add_20_attn | Add |  | 1 × 40960 × 4096 |
| 120 | RMSNorm_20_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 121 | FFN_20 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 122 | Add_20_ffn | Add |  | 1 × 40960 × 4096 |
| 123 | RMSNorm_21_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 124 | Attention_21 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 125 | Add_21_attn | Add |  | 1 × 40960 × 4096 |
| 126 | RMSNorm_21_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 127 | FFN_21 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 128 | Add_21_ffn | Add |  | 1 × 40960 × 4096 |
| 129 | RMSNorm_22_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 130 | Attention_22 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 131 | Add_22_attn | Add |  | 1 × 40960 × 4096 |
| 132 | RMSNorm_22_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 133 | FFN_22 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 134 | Add_22_ffn | Add |  | 1 × 40960 × 4096 |
| 135 | RMSNorm_23_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 136 | Attention_23 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 137 | Add_23_attn | Add |  | 1 × 40960 × 4096 |
| 138 | RMSNorm_23_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 139 | FFN_23 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 140 | Add_23_ffn | Add |  | 1 × 40960 × 4096 |
| 141 | RMSNorm_24_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 142 | Attention_24 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 143 | Add_24_attn | Add |  | 1 × 40960 × 4096 |
| 144 | RMSNorm_24_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 145 | FFN_24 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 146 | Add_24_ffn | Add |  | 1 × 40960 × 4096 |
| 147 | RMSNorm_25_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 148 | Attention_25 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 149 | Add_25_attn | Add |  | 1 × 40960 × 4096 |
| 150 | RMSNorm_25_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 151 | FFN_25 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 152 | Add_25_ffn | Add |  | 1 × 40960 × 4096 |
| 153 | RMSNorm_26_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 154 | Attention_26 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 155 | Add_26_attn | Add |  | 1 × 40960 × 4096 |
| 156 | RMSNorm_26_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 157 | FFN_26 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 158 | Add_26_ffn | Add |  | 1 × 40960 × 4096 |
| 159 | RMSNorm_27_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 160 | Attention_27 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 161 | Add_27_attn | Add |  | 1 × 40960 × 4096 |
| 162 | RMSNorm_27_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 163 | FFN_27 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 164 | Add_27_ffn | Add |  | 1 × 40960 × 4096 |
| 165 | RMSNorm_28_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 166 | Attention_28 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 167 | Add_28_attn | Add |  | 1 × 40960 × 4096 |
| 168 | RMSNorm_28_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 169 | FFN_28 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 170 | Add_28_ffn | Add |  | 1 × 40960 × 4096 |
| 171 | RMSNorm_29_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 172 | Attention_29 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 173 | Add_29_attn | Add |  | 1 × 40960 × 4096 |
| 174 | RMSNorm_29_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 175 | FFN_29 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 176 | Add_29_ffn | Add |  | 1 × 40960 × 4096 |
| 177 | RMSNorm_30_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 178 | Attention_30 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 179 | Add_30_attn | Add |  | 1 × 40960 × 4096 |
| 180 | RMSNorm_30_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 181 | FFN_30 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 182 | Add_30_ffn | Add |  | 1 × 40960 × 4096 |
| 183 | RMSNorm_31_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 184 | Attention_31 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 185 | Add_31_attn | Add |  | 1 × 40960 × 4096 |
| 186 | RMSNorm_31_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 187 | FFN_31 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 188 | Add_31_ffn | Add |  | 1 × 40960 × 4096 |
| 189 | RMSNorm_32_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 190 | Attention_32 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 191 | Add_32_attn | Add |  | 1 × 40960 × 4096 |
| 192 | RMSNorm_32_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 193 | FFN_32 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 194 | Add_32_ffn | Add |  | 1 × 40960 × 4096 |
| 195 | RMSNorm_33_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 196 | Attention_33 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 197 | Add_33_attn | Add |  | 1 × 40960 × 4096 |
| 198 | RMSNorm_33_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 199 | FFN_33 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 200 | Add_33_ffn | Add |  | 1 × 40960 × 4096 |
| 201 | RMSNorm_34_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 202 | Attention_34 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 203 | Add_34_attn | Add |  | 1 × 40960 × 4096 |
| 204 | RMSNorm_34_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 205 | FFN_34 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 206 | Add_34_ffn | Add |  | 1 × 40960 × 4096 |
| 207 | RMSNorm_35_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 208 | Attention_35 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 209 | Add_35_attn | Add |  | 1 × 40960 × 4096 |
| 210 | RMSNorm_35_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 211 | FFN_35 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 212 | Add_35_ffn | Add |  | 1 × 40960 × 4096 |
| 213 | RMSNorm_36_1 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 214 | Attention_36 | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 40960 × 4096 |
| 215 | Add_36_attn | Add |  | 1 × 40960 × 4096 |
| 216 | RMSNorm_36_2 | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 217 | FFN_36 | SwiGLU | embedDim=4096, intermediateSize=12288 | 1 × 40960 × 4096 |
| 218 | Add_36_ffn | Add |  | 1 × 40960 × 4096 |
| 219 | Final_RMSNorm | RMSNorm | normalizedShape=4096 | 1 × 40960 × 4096 |
| 220 | LM_Head | Linear | outFeatures=151936, inFeatures=4096 | 1 × 40960 × 151936 |
| 221 | Output | Output |  | 1 × 40960 × 151936 |

## Verifier findings

- **warn** `attention-no-pe` at `Attention_1`: 36 attention layer(s) present but no positional encoding found. Attention is permutation-invariant, without position information the model cannot distinguish token order. Fix: Add a PositionalEncoding (sinusoidal) or RoPE layer before the first attention layer.
- **info** `deep-attention-default-init` at `Attention_1`: At 36 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class Qwen3_8B(nn.Module):
    def __init__(self):
        super().__init__()

        self.embedding_1 = nn.Embedding(151936, 4096)
        self.rmsNorm_1 = nn.RMSNorm(4096)
        self.groupedQueryAttention_1 = nn.ModuleDict({
            'q_proj': nn.Linear(4096, 4096,        bias=False),   # 32 heads × 128
            'k_proj': nn.Linear(4096, 1024, bias=False),   # 8 KV heads × 128
            'v_proj': nn.Linear(4096, 1024, bias=False),
            'o_proj': nn.Linear(4096, 4096,        bias=False),
        })  # GQA: 32Q / 8KV heads (requires F.scaled_dot_product_attention)
        self.rmsNorm_2 = nn.RMSNorm(4096)
        self.swiglu_1 = nn.ModuleDict({
            'gate_proj': nn.Linear(4096, 12288, bias=False),
            'up_proj':   nn.Linear(4096, 12288, bias=False),
            'down_proj': nn.Linear(12288, 4096, bias=False),
        })  # SwiGLU FFN (LLaMA-style)
        self.rmsNorm_3 = nn.RMSNorm(4096)
        self.groupedQueryAttention_2 = nn.ModuleDict({
            'q_proj': nn.Linear(4096, 4096,        bias=False),   # 32 heads × 128
            'k_proj': nn.Linear(4096, 1024, bias=False),   # 8 KV heads × 128
            'v_proj': nn.Linear(4096, 1024, bias=False),
            'o_proj': nn.Linear(4096, 4096,        bias=False),
        })  # GQA: 32Q / 8KV heads (requires F.scaled_dot_product_attention)
        self.rmsNorm_4 = nn.RMSNorm(4096)
        self.swiglu_2 = nn.ModuleDict({
            'gate_proj': nn.Linear(4096, 12288, bias=False),
            'up_proj':   nn.Linear(4096, 12288, bias=False),
            'down_proj': nn.Linear(12288, 4096, bias=False),
        })  # SwiGLU FFN (LLaMA-style)
        self.rmsNorm_5 = nn.RMSNorm(4096)
        self.groupedQueryAttention_3 = nn.ModuleDict({
            'q_proj': nn.Linear(4096, 4096,        bias=False),   # 32 heads × 128
            'k_proj': nn.Linear(4096, 1024, bias=False),   # 8 KV heads × 128
            'v_proj': nn.Linear(4096, 1024, bias=False),
            'o_proj': nn.Linear(4096, 4096,        bias=False),
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md