# LLaMA-3 Block vs DeepSeek-V3 What a frontier open model adds to the block everyone started from. **DeepSeek-V3 has 666B more parameters than LLaMA-3 Block: 364 layers added, 2 removed, 5 changed.** Source: https://neurarch.com/diff/llama3-block-vs-deepseek-v3.html ## Sides | | LLaMA-3 Block | DeepSeek-V3 | |---|---|---| | Layers | 8 | 370 | | Parameters | 703M | 666B | | Input | 1 × 2048 | 1 × 163840 | | Output | 1 × 2048 × 4096 | 1 × 163840 × 129280 | | Forward-passes | yes | yes | | Est. train cost | $14.27 | $670,780 | | T4 16GB | fits | no | | A100 40GB | fits | no | | H100 80GB | fits | no | ## Deltas (DeepSeek-V3 relative to LLaMA-3 Block) - Parameters: +666B (948× the size) - Layers: +362 - Added 364, removed 2, changed 5, unchanged 3 ## Layer by layer | # | Status | LLaMA-3 Block | Params | Output | DeepSeek-V3 | Params | Output | |---|---|---|---|---|---|---|---| | 1 | changed (shape) | tokens (Input) | | 1 × 2048 | Input (Input) | | 1 × 163840 | | 2 | removed | rope (Rope) | | | | | | | 3 | changed (numEmbeddings, embeddingDim, vocabSize, maxSeqLen) | embed (Embedding) | 525M | 1 × 2048 × 4096 | Embedding (Embedding) | 927M | 1 × 163840 × 7168 | | 4 | added | | | | RoPE (decoupled) (Rope) | | 1 × 163840 × 7168 | | 5 | changed (normalizedShape) | attn_norm (Rms Norm) | 4.1K | 1 × 2048 × 4096 | RMSNorm_1_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 6 | removed | gqa (Grouped Query Attention) | 42M | 1 × 2048 × 4096 | | | | | 7 | added | | | | MLA_1 (Mla) | 110M | 1 × 163840 × 7168 | | 8 | added | | | | Add_1_attn (Add) | | 1 × 163840 × 7168 | | 9 | added | | | | RMSNorm_1_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 10 | added | | | | FFN_1 (Swiglu) | 396M | 1 × 163840 × 7168 | | 11 | added | | | | Add_1_ffn (Add) | | 1 × 163840 × 7168 | | 12 | added | | | | RMSNorm_2_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 13 | added | | | | MLA_2 (Mla) | 110M | 1 × 163840 × 7168 | | 14 | added | | | | Add_2_attn (Add) | | 1 × 163840 × 7168 | | 15 | added | | | | RMSNorm_2_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 16 | added | | | | FFN_2 (Swiglu) | 396M | 1 × 163840 × 7168 | | 17 | added | | | | Add_2_ffn (Add) | | 1 × 163840 × 7168 | | 18 | added | | | | RMSNorm_3_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 19 | added | | | | MLA_3 (Mla) | 110M | 1 × 163840 × 7168 | | 20 | same | residual_1 (Add) | | 1 × 2048 × 4096 | Add_3_attn (Add) | | 1 × 163840 × 7168 | | 21 | changed (normalizedShape) | ffn_norm (Rms Norm) | 4.1K | 1 × 2048 × 4096 | RMSNorm_3_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 22 | changed (embedDim, intermediateSize, dim) | swiglu_ffn (Swiglu) | 135M | 1 × 2048 × 4096 | FFN_3 (Swiglu) | 396M | 1 × 163840 × 7168 | | 23 | added | | | | Add_3_ffn (Add) | | 1 × 163840 × 7168 | | 24 | added | | | | RMSNorm_4_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 25 | added | | | | MLA_4 (Mla) | 110M | 1 × 163840 × 7168 | | 26 | added | | | | Add_4_attn (Add) | | 1 × 163840 × 7168 | | 27 | added | | | | RMSNorm_4_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 28 | added | | | | MoE_4 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 29 | added | | | | Add_4_ffn (Add) | | 1 × 163840 × 7168 | | 30 | added | | | | RMSNorm_5_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 31 | added | | | | MLA_5 (Mla) | 110M | 1 × 163840 × 7168 | | 32 | added | | | | Add_5_attn (Add) | | 1 × 163840 × 7168 | | 33 | added | | | | RMSNorm_5_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 34 | added | | | | MoE_5 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 35 | added | | | | Add_5_ffn (Add) | | 1 × 163840 × 7168 | | 36 | added | | | | RMSNorm_6_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 37 | added | | | | MLA_6 (Mla) | 110M | 1 × 163840 × 7168 | | 38 | added | | | | Add_6_attn (Add) | | 1 × 163840 × 7168 | | 39 | added | | | | RMSNorm_6_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 40 | added | | | | MoE_6 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 41 | added | | | | Add_6_ffn (Add) | | 1 × 163840 × 7168 | | 42 | added | | | | RMSNorm_7_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 43 | added | | | | MLA_7 (Mla) | 110M | 1 × 163840 × 7168 | | 44 | added | | | | Add_7_attn (Add) | | 1 × 163840 × 7168 | | 45 | added | | | | RMSNorm_7_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 46 | added | | | | MoE_7 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 47 | added | | | | Add_7_ffn (Add) | | 1 × 163840 × 7168 | | 48 | added | | | | RMSNorm_8_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 49 | added | | | | MLA_8 (Mla) | 110M | 1 × 163840 × 7168 | | 50 | added | | | | Add_8_attn (Add) | | 1 × 163840 × 7168 | | 51 | added | | | | RMSNorm_8_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 52 | added | | | | MoE_8 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 53 | added | | | | Add_8_ffn (Add) | | 1 × 163840 × 7168 | | 54 | added | | | | RMSNorm_9_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 55 | added | | | | MLA_9 (Mla) | 110M | 1 × 163840 × 7168 | | 56 | added | | | | Add_9_attn (Add) | | 1 × 163840 × 7168 | | 57 | added | | | | RMSNorm_9_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 58 | added | | | | MoE_9 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 59 | added | | | | Add_9_ffn (Add) | | 1 × 163840 × 7168 | | 60 | added | | | | RMSNorm_10_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 61 | added | | | | MLA_10 (Mla) | 110M | 1 × 163840 × 7168 | | 62 | added | | | | Add_10_attn (Add) | | 1 × 163840 × 7168 | | 63 | added | | | | RMSNorm_10_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 64 | added | | | | MoE_10 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 65 | added | | | | Add_10_ffn (Add) | | 1 × 163840 × 7168 | | 66 | added | | | | RMSNorm_11_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 67 | added | | | | MLA_11 (Mla) | 110M | 1 × 163840 × 7168 | | 68 | added | | | | Add_11_attn (Add) | | 1 × 163840 × 7168 | | 69 | added | | | | RMSNorm_11_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 70 | added | | | | MoE_11 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 71 | added | | | | Add_11_ffn (Add) | | 1 × 163840 × 7168 | | 72 | added | | | | RMSNorm_12_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 73 | added | | | | MLA_12 (Mla) | 110M | 1 × 163840 × 7168 | | 74 | added | | | | Add_12_attn (Add) | | 1 × 163840 × 7168 | | 75 | added | | | | RMSNorm_12_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 76 | added | | | | MoE_12 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 77 | added | | | | Add_12_ffn (Add) | | 1 × 163840 × 7168 | | 78 | added | | | | RMSNorm_13_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 79 | added | | | | MLA_13 (Mla) | 110M | 1 × 163840 × 7168 | | 80 | added | | | | Add_13_attn (Add) | | 1 × 163840 × 7168 | | 81 | added | | | | RMSNorm_13_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 82 | added | | | | MoE_13 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 83 | added | | | | Add_13_ffn (Add) | | 1 × 163840 × 7168 | | 84 | added | | | | RMSNorm_14_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 85 | added | | | | MLA_14 (Mla) | 110M | 1 × 163840 × 7168 | | 86 | added | | | | Add_14_attn (Add) | | 1 × 163840 × 7168 | | 87 | added | | | | RMSNorm_14_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 88 | added | | | | MoE_14 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 89 | added | | | | Add_14_ffn (Add) | | 1 × 163840 × 7168 | | 90 | added | | | | RMSNorm_15_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 91 | added | | | | MLA_15 (Mla) | 110M | 1 × 163840 × 7168 | | 92 | added | | | | Add_15_attn (Add) | | 1 × 163840 × 7168 | | 93 | added | | | | RMSNorm_15_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 94 | added | | | | MoE_15 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 95 | added | | | | Add_15_ffn (Add) | | 1 × 163840 × 7168 | | 96 | added | | | | RMSNorm_16_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 97 | added | | | | MLA_16 (Mla) | 110M | 1 × 163840 × 7168 | | 98 | added | | | | Add_16_attn (Add) | | 1 × 163840 × 7168 | | 99 | added | | | | RMSNorm_16_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 100 | added | | | | MoE_16 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 101 | added | | | | Add_16_ffn (Add) | | 1 × 163840 × 7168 | | 102 | added | | | | RMSNorm_17_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 103 | added | | | | MLA_17 (Mla) | 110M | 1 × 163840 × 7168 | | 104 | added | | | | Add_17_attn (Add) | | 1 × 163840 × 7168 | | 105 | added | | | | RMSNorm_17_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 106 | added | | | | MoE_17 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 107 | added | | | | Add_17_ffn (Add) | | 1 × 163840 × 7168 | | 108 | added | | | | RMSNorm_18_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 109 | added | | | | MLA_18 (Mla) | 110M | 1 × 163840 × 7168 | | 110 | added | | | | Add_18_attn (Add) | | 1 × 163840 × 7168 | | 111 | added | | | | RMSNorm_18_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 112 | added | | | | MoE_18 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 113 | added | | | | Add_18_ffn (Add) | | 1 × 163840 × 7168 | | 114 | added | | | | RMSNorm_19_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 115 | added | | | | MLA_19 (Mla) | 110M | 1 × 163840 × 7168 | | 116 | added | | | | Add_19_attn (Add) | | 1 × 163840 × 7168 | | 117 | added | | | | RMSNorm_19_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 118 | added | | | | MoE_19 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 119 | added | | | | Add_19_ffn (Add) | | 1 × 163840 × 7168 | | 120 | added | | | | RMSNorm_20_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 121 | added | | | | MLA_20 (Mla) | 110M | 1 × 163840 × 7168 | | 122 | added | | | | Add_20_attn (Add) | | 1 × 163840 × 7168 | | 123 | added | | | | RMSNorm_20_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 124 | added | | | | MoE_20 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 125 | added | | | | Add_20_ffn (Add) | | 1 × 163840 × 7168 | | 126 | added | | | | RMSNorm_21_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 127 | added | | | | MLA_21 (Mla) | 110M | 1 × 163840 × 7168 | | 128 | added | | | | Add_21_attn (Add) | | 1 × 163840 × 7168 | | 129 | added | | | | RMSNorm_21_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 130 | added | | | | MoE_21 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 131 | added | | | | Add_21_ffn (Add) | | 1 × 163840 × 7168 | | 132 | added | | | | RMSNorm_22_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 133 | added | | | | MLA_22 (Mla) | 110M | 1 × 163840 × 7168 | | 134 | added | | | | Add_22_attn (Add) | | 1 × 163840 × 7168 | | 135 | added | | | | RMSNorm_22_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 136 | added | | | | MoE_22 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 137 | added | | | | Add_22_ffn (Add) | | 1 × 163840 × 7168 | | 138 | added | | | | RMSNorm_23_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 139 | added | | | | MLA_23 (Mla) | 110M | 1 × 163840 × 7168 | | 140 | added | | | | Add_23_attn (Add) | | 1 × 163840 × 7168 | | 141 | added | | | | RMSNorm_23_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 142 | added | | | | MoE_23 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 143 | added | | | | Add_23_ffn (Add) | | 1 × 163840 × 7168 | | 144 | added | | | | RMSNorm_24_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 145 | added | | | | MLA_24 (Mla) | 110M | 1 × 163840 × 7168 | | 146 | added | | | | Add_24_attn (Add) | | 1 × 163840 × 7168 | | 147 | added | | | | RMSNorm_24_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 148 | added | | | | MoE_24 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 149 | added | | | | Add_24_ffn (Add) | | 1 × 163840 × 7168 | | 150 | added | | | | RMSNorm_25_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 151 | added | | | | MLA_25 (Mla) | 110M | 1 × 163840 × 7168 | | 152 | added | | | | Add_25_attn (Add) | | 1 × 163840 × 7168 | | 153 | added | | | | RMSNorm_25_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 154 | added | | | | MoE_25 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 155 | added | | | | Add_25_ffn (Add) | | 1 × 163840 × 7168 | | 156 | added | | | | RMSNorm_26_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 157 | added | | | | MLA_26 (Mla) | 110M | 1 × 163840 × 7168 | | 158 | added | | | | Add_26_attn (Add) | | 1 × 163840 × 7168 | | 159 | added | | | | RMSNorm_26_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 160 | added | | | | MoE_26 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 161 | added | | | | Add_26_ffn (Add) | | 1 × 163840 × 7168 | | 162 | added | | | | RMSNorm_27_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 163 | added | | | | MLA_27 (Mla) | 110M | 1 × 163840 × 7168 | | 164 | added | | | | Add_27_attn (Add) | | 1 × 163840 × 7168 | | 165 | added | | | | RMSNorm_27_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 166 | added | | | | MoE_27 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 167 | added | | | | Add_27_ffn (Add) | | 1 × 163840 × 7168 | | 168 | added | | | | RMSNorm_28_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 169 | added | | | | MLA_28 (Mla) | 110M | 1 × 163840 × 7168 | | 170 | added | | | | Add_28_attn (Add) | | 1 × 163840 × 7168 | | 171 | added | | | | RMSNorm_28_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 172 | added | | | | MoE_28 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 173 | added | | | | Add_28_ffn (Add) | | 1 × 163840 × 7168 | | 174 | added | | | | RMSNorm_29_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 175 | added | | | | MLA_29 (Mla) | 110M | 1 × 163840 × 7168 | | 176 | added | | | | Add_29_attn (Add) | | 1 × 163840 × 7168 | | 177 | added | | | | RMSNorm_29_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 178 | added | | | | MoE_29 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 179 | added | | | | Add_29_ffn (Add) | | 1 × 163840 × 7168 | | 180 | added | | | | RMSNorm_30_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 181 | added | | | | MLA_30 (Mla) | 110M | 1 × 163840 × 7168 | | 182 | added | | | | Add_30_attn (Add) | | 1 × 163840 × 7168 | | 183 | added | | | | RMSNorm_30_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 184 | added | | | | MoE_30 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 185 | added | | | | Add_30_ffn (Add) | | 1 × 163840 × 7168 | | 186 | added | | | | RMSNorm_31_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 187 | added | | | | MLA_31 (Mla) | 110M | 1 × 163840 × 7168 | | 188 | added | | | | Add_31_attn (Add) | | 1 × 163840 × 7168 | | 189 | added | | | | RMSNorm_31_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 190 | added | | | | MoE_31 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 191 | added | | | | Add_31_ffn (Add) | | 1 × 163840 × 7168 | | 192 | added | | | | RMSNorm_32_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 193 | added | | | | MLA_32 (Mla) | 110M | 1 × 163840 × 7168 | | 194 | added | | | | Add_32_attn (Add) | | 1 × 163840 × 7168 | | 195 | added | | | | RMSNorm_32_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 196 | added | | | | MoE_32 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 197 | added | | | | Add_32_ffn (Add) | | 1 × 163840 × 7168 | | 198 | added | | | | RMSNorm_33_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 199 | added | | | | MLA_33 (Mla) | 110M | 1 × 163840 × 7168 | | 200 | added | | | | Add_33_attn (Add) | | 1 × 163840 × 7168 | | 201 | added | | | | RMSNorm_33_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 202 | added | | | | MoE_33 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 203 | added | | | | Add_33_ffn (Add) | | 1 × 163840 × 7168 | | 204 | added | | | | RMSNorm_34_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 205 | added | | | | MLA_34 (Mla) | 110M | 1 × 163840 × 7168 | | 206 | added | | | | Add_34_attn (Add) | | 1 × 163840 × 7168 | | 207 | added | | | | RMSNorm_34_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 208 | added | | | | MoE_34 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 209 | added | | | | Add_34_ffn (Add) | | 1 × 163840 × 7168 | | 210 | added | | | | RMSNorm_35_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 211 | added | | | | MLA_35 (Mla) | 110M | 1 × 163840 × 7168 | | 212 | added | | | | Add_35_attn (Add) | | 1 × 163840 × 7168 | | 213 | added | | | | RMSNorm_35_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 214 | added | | | | MoE_35 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 215 | added | | | | Add_35_ffn (Add) | | 1 × 163840 × 7168 | | 216 | added | | | | RMSNorm_36_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 217 | added | | | | MLA_36 (Mla) | 110M | 1 × 163840 × 7168 | | 218 | added | | | | Add_36_attn (Add) | | 1 × 163840 × 7168 | | 219 | added | | | | RMSNorm_36_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 220 | added | | | | MoE_36 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 221 | added | | | | Add_36_ffn (Add) | | 1 × 163840 × 7168 | | 222 | added | | | | RMSNorm_37_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 223 | added | | | | MLA_37 (Mla) | 110M | 1 × 163840 × 7168 | | 224 | added | | | | Add_37_attn (Add) | | 1 × 163840 × 7168 | | 225 | added | | | | RMSNorm_37_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 226 | added | | | | MoE_37 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 227 | added | | | | Add_37_ffn (Add) | | 1 × 163840 × 7168 | | 228 | added | | | | RMSNorm_38_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 229 | added | | | | MLA_38 (Mla) | 110M | 1 × 163840 × 7168 | | 230 | added | | | | Add_38_attn (Add) | | 1 × 163840 × 7168 | | 231 | added | | | | RMSNorm_38_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 232 | added | | | | MoE_38 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 233 | added | | | | Add_38_ffn (Add) | | 1 × 163840 × 7168 | | 234 | added | | | | RMSNorm_39_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 235 | added | | | | MLA_39 (Mla) | 110M | 1 × 163840 × 7168 | | 236 | added | | | | Add_39_attn (Add) | | 1 × 163840 × 7168 | | 237 | added | | | | RMSNorm_39_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 238 | added | | | | MoE_39 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 239 | added | | | | Add_39_ffn (Add) | | 1 × 163840 × 7168 | | 240 | added | | | | RMSNorm_40_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 241 | added | | | | MLA_40 (Mla) | 110M | 1 × 163840 × 7168 | | 242 | added | | | | Add_40_attn (Add) | | 1 × 163840 × 7168 | | 243 | added | | | | RMSNorm_40_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 244 | added | | | | MoE_40 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 245 | added | | | | Add_40_ffn (Add) | | 1 × 163840 × 7168 | | 246 | added | | | | RMSNorm_41_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 247 | added | | | | MLA_41 (Mla) | 110M | 1 × 163840 × 7168 | | 248 | added | | | | Add_41_attn (Add) | | 1 × 163840 × 7168 | | 249 | added | | | | RMSNorm_41_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 250 | added | | | | MoE_41 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 251 | added | | | | Add_41_ffn (Add) | | 1 × 163840 × 7168 | | 252 | added | | | | RMSNorm_42_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 253 | added | | | | MLA_42 (Mla) | 110M | 1 × 163840 × 7168 | | 254 | added | | | | Add_42_attn (Add) | | 1 × 163840 × 7168 | | 255 | added | | | | RMSNorm_42_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 256 | added | | | | MoE_42 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 257 | added | | | | Add_42_ffn (Add) | | 1 × 163840 × 7168 | | 258 | added | | | | RMSNorm_43_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 259 | added | | | | MLA_43 (Mla) | 110M | 1 × 163840 × 7168 | | 260 | added | | | | Add_43_attn (Add) | | 1 × 163840 × 7168 | | 261 | added | | | | RMSNorm_43_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 262 | added | | | | MoE_43 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 263 | added | | | | Add_43_ffn (Add) | | 1 × 163840 × 7168 | | 264 | added | | | | RMSNorm_44_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 265 | added | | | | MLA_44 (Mla) | 110M | 1 × 163840 × 7168 | | 266 | added | | | | Add_44_attn (Add) | | 1 × 163840 × 7168 | | 267 | added | | | | RMSNorm_44_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 268 | added | | | | MoE_44 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 269 | added | | | | Add_44_ffn (Add) | | 1 × 163840 × 7168 | | 270 | added | | | | RMSNorm_45_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 271 | added | | | | MLA_45 (Mla) | 110M | 1 × 163840 × 7168 | | 272 | added | | | | Add_45_attn (Add) | | 1 × 163840 × 7168 | | 273 | added | | | | RMSNorm_45_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 274 | added | | | | MoE_45 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 275 | added | | | | Add_45_ffn (Add) | | 1 × 163840 × 7168 | | 276 | added | | | | RMSNorm_46_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 277 | added | | | | MLA_46 (Mla) | 110M | 1 × 163840 × 7168 | | 278 | added | | | | Add_46_attn (Add) | | 1 × 163840 × 7168 | | 279 | added | | | | RMSNorm_46_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 280 | added | | | | MoE_46 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 281 | added | | | | Add_46_ffn (Add) | | 1 × 163840 × 7168 | | 282 | added | | | | RMSNorm_47_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 283 | added | | | | MLA_47 (Mla) | 110M | 1 × 163840 × 7168 | | 284 | added | | | | Add_47_attn (Add) | | 1 × 163840 × 7168 | | 285 | added | | | | RMSNorm_47_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 286 | added | | | | MoE_47 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 287 | added | | | | Add_47_ffn (Add) | | 1 × 163840 × 7168 | | 288 | added | | | | RMSNorm_48_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 289 | added | | | | MLA_48 (Mla) | 110M | 1 × 163840 × 7168 | | 290 | added | | | | Add_48_attn (Add) | | 1 × 163840 × 7168 | | 291 | added | | | | RMSNorm_48_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 292 | added | | | | MoE_48 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 293 | added | | | | Add_48_ffn (Add) | | 1 × 163840 × 7168 | | 294 | added | | | | RMSNorm_49_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 295 | added | | | | MLA_49 (Mla) | 110M | 1 × 163840 × 7168 | | 296 | added | | | | Add_49_attn (Add) | | 1 × 163840 × 7168 | | 297 | added | | | | RMSNorm_49_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 298 | added | | | | MoE_49 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 299 | added | | | | Add_49_ffn (Add) | | 1 × 163840 × 7168 | | 300 | added | | | | RMSNorm_50_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 301 | added | | | | MLA_50 (Mla) | 110M | 1 × 163840 × 7168 | | 302 | added | | | | Add_50_attn (Add) | | 1 × 163840 × 7168 | | 303 | added | | | | RMSNorm_50_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 304 | added | | | | MoE_50 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 305 | added | | | | Add_50_ffn (Add) | | 1 × 163840 × 7168 | | 306 | added | | | | RMSNorm_51_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 307 | added | | | | MLA_51 (Mla) | 110M | 1 × 163840 × 7168 | | 308 | added | | | | Add_51_attn (Add) | | 1 × 163840 × 7168 | | 309 | added | | | | RMSNorm_51_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 310 | added | | | | MoE_51 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 311 | added | | | | Add_51_ffn (Add) | | 1 × 163840 × 7168 | | 312 | added | | | | RMSNorm_52_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 313 | added | | | | MLA_52 (Mla) | 110M | 1 × 163840 × 7168 | | 314 | added | | | | Add_52_attn (Add) | | 1 × 163840 × 7168 | | 315 | added | | | | RMSNorm_52_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 316 | added | | | | MoE_52 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 317 | added | | | | Add_52_ffn (Add) | | 1 × 163840 × 7168 | | 318 | added | | | | RMSNorm_53_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 319 | added | | | | MLA_53 (Mla) | 110M | 1 × 163840 × 7168 | | 320 | added | | | | Add_53_attn (Add) | | 1 × 163840 × 7168 | | 321 | added | | | | RMSNorm_53_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 322 | added | | | | MoE_53 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 323 | added | | | | Add_53_ffn (Add) | | 1 × 163840 × 7168 | | 324 | added | | | | RMSNorm_54_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 325 | added | | | | MLA_54 (Mla) | 110M | 1 × 163840 × 7168 | | 326 | added | | | | Add_54_attn (Add) | | 1 × 163840 × 7168 | | 327 | added | | | | RMSNorm_54_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 328 | added | | | | MoE_54 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 329 | added | | | | Add_54_ffn (Add) | | 1 × 163840 × 7168 | | 330 | added | | | | RMSNorm_55_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 331 | added | | | | MLA_55 (Mla) | 110M | 1 × 163840 × 7168 | | 332 | added | | | | Add_55_attn (Add) | | 1 × 163840 × 7168 | | 333 | added | | | | RMSNorm_55_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 334 | added | | | | MoE_55 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 335 | added | | | | Add_55_ffn (Add) | | 1 × 163840 × 7168 | | 336 | added | | | | RMSNorm_56_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 337 | added | | | | MLA_56 (Mla) | 110M | 1 × 163840 × 7168 | | 338 | added | | | | Add_56_attn (Add) | | 1 × 163840 × 7168 | | 339 | added | | | | RMSNorm_56_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 340 | added | | | | MoE_56 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 341 | added | | | | Add_56_ffn (Add) | | 1 × 163840 × 7168 | | 342 | added | | | | RMSNorm_57_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 343 | added | | | | MLA_57 (Mla) | 110M | 1 × 163840 × 7168 | | 344 | added | | | | Add_57_attn (Add) | | 1 × 163840 × 7168 | | 345 | added | | | | RMSNorm_57_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 346 | added | | | | MoE_57 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 347 | added | | | | Add_57_ffn (Add) | | 1 × 163840 × 7168 | | 348 | added | | | | RMSNorm_58_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 349 | added | | | | MLA_58 (Mla) | 110M | 1 × 163840 × 7168 | | 350 | added | | | | Add_58_attn (Add) | | 1 × 163840 × 7168 | | 351 | added | | | | RMSNorm_58_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 352 | added | | | | MoE_58 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 353 | added | | | | Add_58_ffn (Add) | | 1 × 163840 × 7168 | | 354 | added | | | | RMSNorm_59_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 355 | added | | | | MLA_59 (Mla) | 110M | 1 × 163840 × 7168 | | 356 | added | | | | Add_59_attn (Add) | | 1 × 163840 × 7168 | | 357 | added | | | | RMSNorm_59_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 358 | added | | | | MoE_59 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 359 | added | | | | Add_59_ffn (Add) | | 1 × 163840 × 7168 | | 360 | added | | | | RMSNorm_60_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 361 | added | | | | MLA_60 (Mla) | 110M | 1 × 163840 × 7168 | | 362 | added | | | | Add_60_attn (Add) | | 1 × 163840 × 7168 | | 363 | added | | | | RMSNorm_60_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 364 | added | | | | MoE_60 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 365 | added | | | | Add_60_ffn (Add) | | 1 × 163840 × 7168 | | 366 | added | | | | RMSNorm_61_1 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 367 | added | | | | MLA_61 (Mla) | 110M | 1 × 163840 × 7168 | | 368 | added | | | | Add_61_attn (Add) | | 1 × 163840 × 7168 | | 369 | added | | | | RMSNorm_61_2 (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 370 | added | | | | MoE_61 (Shared Expert Mo E) | 11B | 1 × 163840 × 7168 | | 371 | same | residual_2 (Add) | | 1 × 2048 × 4096 | Add_61_ffn (Add) | | 1 × 163840 × 7168 | | 372 | added | | | | Final_RMSNorm (Rms Norm) | 7.2K | 1 × 163840 × 7168 | | 373 | added | | | | LM_Head (Linear) | 927M | 1 × 163840 × 129280 | | 374 | same | hidden_state (Output) | | 1 × 2048 × 4096 | Output (Output) | | 1 × 163840 × 129280 | ## What this is not - The two are priced at different declared inputs (1 × 2048 against 1 × 163840), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected. - Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not. - Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run. ## Graphs - LLaMA-3 Block: https://neurarch.com/templates/llama3-block/model.json - DeepSeek-V3: https://neurarch.com/templates/deepseek-v3/model.json - Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`