# MiMo-V2.5

Hugging Face: `XiaomiMiMo/MiMo-V2.5`
Reconstructed from `config.json` alone. No weights were read.

| | |
|---|---|
| Derived from structure | 315,696,050,688 |
| In the published checkpoint | 310,775,040,000 (safetensors.total, 2026-07-09) |
| Delta | +1.58% |
| Comparability | quantized |
| Layers | 300 |
| Priced on | A10G (24GB) |
| Est. one run | $24900539.68 |

> This checkpoint is stored quantized (fp8). The tensor count in the file counts stored elements under a packing scheme, not logical parameters, so the two numbers below are not measuring the same thing in either direction.

## Structure

| # | Layer | Type | Output shape |
|---|---|---|---|
| 1 | Input | Input | 1 × 1048576 |
| 2 | Embedding | Embedding | 1 × 1048576 × 4096 |
| 3 | RoPE | RoPE | 1 × 1048576 × 4096 |
| 4 | Vision input | Input | 3 × 224 × 224 |
| 5 | PatchEmbed | Patch Embed | 196 × 1280 |
| 6 | Patch_Position_Embedding | Learned Pos Embed | 196 × 1280 |
| 7 | Vision encoder (internals not in config) | Projection | 196 × 4096 |
| 8 | Vision tokens | Reshape | 1 × 196 × 4096 |
| 9 | Audio input | Input | 1 × 128 × 1 |
| 10 | Audio feature embed | Projection | 1 × 128 × 4096 |
| 11 | Audio encoder (internals not in config) | Projection | 1 × 128 × 4096 |
| 12 | Audio tokens | Reshape | 1 × 256 × 4096 |
| 13 | Multimodal fusion (concat tokens) | Concatenate | 1 × 1049028 × 4096 |
| 14 | RMSNorm_1_1 | RMSNorm | 1 × 1049028 × 4096 |
| 15 | Attention_1 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 16 | Add_1_attn | Add | 1 × 1049028 × 4096 |
| 17 | RMSNorm_1_2 | RMSNorm | 1 × 1049028 × 4096 |
| 18 | MoE_1 | MoE Layer | 1 × 1049028 × 4096 |
| 19 | Add_1_ffn | Add | 1 × 1049028 × 4096 |
| 20 | RMSNorm_2_1 | RMSNorm | 1 × 1049028 × 4096 |
| 21 | Attention_2 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 22 | Add_2_attn | Add | 1 × 1049028 × 4096 |
| 23 | RMSNorm_2_2 | RMSNorm | 1 × 1049028 × 4096 |
| 24 | MoE_2 | MoE Layer | 1 × 1049028 × 4096 |
| 25 | Add_2_ffn | Add | 1 × 1049028 × 4096 |
| 26 | RMSNorm_3_1 | RMSNorm | 1 × 1049028 × 4096 |
| 27 | Attention_3 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 28 | Add_3_attn | Add | 1 × 1049028 × 4096 |
| 29 | RMSNorm_3_2 | RMSNorm | 1 × 1049028 × 4096 |
| 30 | MoE_3 | MoE Layer | 1 × 1049028 × 4096 |
| 31 | Add_3_ffn | Add | 1 × 1049028 × 4096 |
| 32 | RMSNorm_4_1 | RMSNorm | 1 × 1049028 × 4096 |
| 33 | Attention_4 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 34 | Add_4_attn | Add | 1 × 1049028 × 4096 |
| 35 | RMSNorm_4_2 | RMSNorm | 1 × 1049028 × 4096 |
| 36 | MoE_4 | MoE Layer | 1 × 1049028 × 4096 |
| 37 | Add_4_ffn | Add | 1 × 1049028 × 4096 |
| 38 | RMSNorm_5_1 | RMSNorm | 1 × 1049028 × 4096 |
| 39 | Attention_5 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 40 | Add_5_attn | Add | 1 × 1049028 × 4096 |
| 41 | RMSNorm_5_2 | RMSNorm | 1 × 1049028 × 4096 |
| 42 | MoE_5 | MoE Layer | 1 × 1049028 × 4096 |
| 43 | Add_5_ffn | Add | 1 × 1049028 × 4096 |
| 44 | RMSNorm_6_1 | RMSNorm | 1 × 1049028 × 4096 |
| 45 | Attention_6 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 46 | Add_6_attn | Add | 1 × 1049028 × 4096 |
| 47 | RMSNorm_6_2 | RMSNorm | 1 × 1049028 × 4096 |
| 48 | MoE_6 | MoE Layer | 1 × 1049028 × 4096 |
| 49 | Add_6_ffn | Add | 1 × 1049028 × 4096 |
| 50 | RMSNorm_7_1 | RMSNorm | 1 × 1049028 × 4096 |
| 51 | Attention_7 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 52 | Add_7_attn | Add | 1 × 1049028 × 4096 |
| 53 | RMSNorm_7_2 | RMSNorm | 1 × 1049028 × 4096 |
| 54 | MoE_7 | MoE Layer | 1 × 1049028 × 4096 |
| 55 | Add_7_ffn | Add | 1 × 1049028 × 4096 |
| 56 | RMSNorm_8_1 | RMSNorm | 1 × 1049028 × 4096 |
| 57 | Attention_8 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 58 | Add_8_attn | Add | 1 × 1049028 × 4096 |
| 59 | RMSNorm_8_2 | RMSNorm | 1 × 1049028 × 4096 |
| 60 | MoE_8 | MoE Layer | 1 × 1049028 × 4096 |
| 61 | Add_8_ffn | Add | 1 × 1049028 × 4096 |
| 62 | RMSNorm_9_1 | RMSNorm | 1 × 1049028 × 4096 |
| 63 | Attention_9 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 64 | Add_9_attn | Add | 1 × 1049028 × 4096 |
| 65 | RMSNorm_9_2 | RMSNorm | 1 × 1049028 × 4096 |
| 66 | MoE_9 | MoE Layer | 1 × 1049028 × 4096 |
| 67 | Add_9_ffn | Add | 1 × 1049028 × 4096 |
| 68 | RMSNorm_10_1 | RMSNorm | 1 × 1049028 × 4096 |
| 69 | Attention_10 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 70 | Add_10_attn | Add | 1 × 1049028 × 4096 |
| 71 | RMSNorm_10_2 | RMSNorm | 1 × 1049028 × 4096 |
| 72 | MoE_10 | MoE Layer | 1 × 1049028 × 4096 |
| 73 | Add_10_ffn | Add | 1 × 1049028 × 4096 |
| 74 | RMSNorm_11_1 | RMSNorm | 1 × 1049028 × 4096 |
| 75 | Attention_11 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 76 | Add_11_attn | Add | 1 × 1049028 × 4096 |
| 77 | RMSNorm_11_2 | RMSNorm | 1 × 1049028 × 4096 |
| 78 | MoE_11 | MoE Layer | 1 × 1049028 × 4096 |
| 79 | Add_11_ffn | Add | 1 × 1049028 × 4096 |
| 80 | RMSNorm_12_1 | RMSNorm | 1 × 1049028 × 4096 |
| 81 | Attention_12 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 82 | Add_12_attn | Add | 1 × 1049028 × 4096 |
| 83 | RMSNorm_12_2 | RMSNorm | 1 × 1049028 × 4096 |
| 84 | MoE_12 | MoE Layer | 1 × 1049028 × 4096 |
| 85 | Add_12_ffn | Add | 1 × 1049028 × 4096 |
| 86 | RMSNorm_13_1 | RMSNorm | 1 × 1049028 × 4096 |
| 87 | Attention_13 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 88 | Add_13_attn | Add | 1 × 1049028 × 4096 |
| 89 | RMSNorm_13_2 | RMSNorm | 1 × 1049028 × 4096 |
| 90 | MoE_13 | MoE Layer | 1 × 1049028 × 4096 |
| 91 | Add_13_ffn | Add | 1 × 1049028 × 4096 |
| 92 | RMSNorm_14_1 | RMSNorm | 1 × 1049028 × 4096 |
| 93 | Attention_14 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 94 | Add_14_attn | Add | 1 × 1049028 × 4096 |
| 95 | RMSNorm_14_2 | RMSNorm | 1 × 1049028 × 4096 |
| 96 | MoE_14 | MoE Layer | 1 × 1049028 × 4096 |
| 97 | Add_14_ffn | Add | 1 × 1049028 × 4096 |
| 98 | RMSNorm_15_1 | RMSNorm | 1 × 1049028 × 4096 |
| 99 | Attention_15 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 100 | Add_15_attn | Add | 1 × 1049028 × 4096 |
| 101 | RMSNorm_15_2 | RMSNorm | 1 × 1049028 × 4096 |
| 102 | MoE_15 | MoE Layer | 1 × 1049028 × 4096 |
| 103 | Add_15_ffn | Add | 1 × 1049028 × 4096 |
| 104 | RMSNorm_16_1 | RMSNorm | 1 × 1049028 × 4096 |
| 105 | Attention_16 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 106 | Add_16_attn | Add | 1 × 1049028 × 4096 |
| 107 | RMSNorm_16_2 | RMSNorm | 1 × 1049028 × 4096 |
| 108 | MoE_16 | MoE Layer | 1 × 1049028 × 4096 |
| 109 | Add_16_ffn | Add | 1 × 1049028 × 4096 |
| 110 | RMSNorm_17_1 | RMSNorm | 1 × 1049028 × 4096 |
| 111 | Attention_17 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 112 | Add_17_attn | Add | 1 × 1049028 × 4096 |
| 113 | RMSNorm_17_2 | RMSNorm | 1 × 1049028 × 4096 |
| 114 | MoE_17 | MoE Layer | 1 × 1049028 × 4096 |
| 115 | Add_17_ffn | Add | 1 × 1049028 × 4096 |
| 116 | RMSNorm_18_1 | RMSNorm | 1 × 1049028 × 4096 |
| 117 | Attention_18 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 118 | Add_18_attn | Add | 1 × 1049028 × 4096 |
| 119 | RMSNorm_18_2 | RMSNorm | 1 × 1049028 × 4096 |
| 120 | MoE_18 | MoE Layer | 1 × 1049028 × 4096 |
| 121 | Add_18_ffn | Add | 1 × 1049028 × 4096 |
| 122 | RMSNorm_19_1 | RMSNorm | 1 × 1049028 × 4096 |
| 123 | Attention_19 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 124 | Add_19_attn | Add | 1 × 1049028 × 4096 |
| 125 | RMSNorm_19_2 | RMSNorm | 1 × 1049028 × 4096 |
| 126 | MoE_19 | MoE Layer | 1 × 1049028 × 4096 |
| 127 | Add_19_ffn | Add | 1 × 1049028 × 4096 |
| 128 | RMSNorm_20_1 | RMSNorm | 1 × 1049028 × 4096 |
| 129 | Attention_20 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 130 | Add_20_attn | Add | 1 × 1049028 × 4096 |
| 131 | RMSNorm_20_2 | RMSNorm | 1 × 1049028 × 4096 |
| 132 | MoE_20 | MoE Layer | 1 × 1049028 × 4096 |
| 133 | Add_20_ffn | Add | 1 × 1049028 × 4096 |
| 134 | RMSNorm_21_1 | RMSNorm | 1 × 1049028 × 4096 |
| 135 | Attention_21 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 136 | Add_21_attn | Add | 1 × 1049028 × 4096 |
| 137 | RMSNorm_21_2 | RMSNorm | 1 × 1049028 × 4096 |
| 138 | MoE_21 | MoE Layer | 1 × 1049028 × 4096 |
| 139 | Add_21_ffn | Add | 1 × 1049028 × 4096 |
| 140 | RMSNorm_22_1 | RMSNorm | 1 × 1049028 × 4096 |
| 141 | Attention_22 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 142 | Add_22_attn | Add | 1 × 1049028 × 4096 |
| 143 | RMSNorm_22_2 | RMSNorm | 1 × 1049028 × 4096 |
| 144 | MoE_22 | MoE Layer | 1 × 1049028 × 4096 |
| 145 | Add_22_ffn | Add | 1 × 1049028 × 4096 |
| 146 | RMSNorm_23_1 | RMSNorm | 1 × 1049028 × 4096 |
| 147 | Attention_23 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 148 | Add_23_attn | Add | 1 × 1049028 × 4096 |
| 149 | RMSNorm_23_2 | RMSNorm | 1 × 1049028 × 4096 |
| 150 | MoE_23 | MoE Layer | 1 × 1049028 × 4096 |
| 151 | Add_23_ffn | Add | 1 × 1049028 × 4096 |
| 152 | RMSNorm_24_1 | RMSNorm | 1 × 1049028 × 4096 |
| 153 | Attention_24 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 154 | Add_24_attn | Add | 1 × 1049028 × 4096 |
| 155 | RMSNorm_24_2 | RMSNorm | 1 × 1049028 × 4096 |
| 156 | MoE_24 | MoE Layer | 1 × 1049028 × 4096 |
| 157 | Add_24_ffn | Add | 1 × 1049028 × 4096 |
| 158 | RMSNorm_25_1 | RMSNorm | 1 × 1049028 × 4096 |
| 159 | Attention_25 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 160 | Add_25_attn | Add | 1 × 1049028 × 4096 |
| 161 | RMSNorm_25_2 | RMSNorm | 1 × 1049028 × 4096 |
| 162 | MoE_25 | MoE Layer | 1 × 1049028 × 4096 |
| 163 | Add_25_ffn | Add | 1 × 1049028 × 4096 |
| 164 | RMSNorm_26_1 | RMSNorm | 1 × 1049028 × 4096 |
| 165 | Attention_26 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 166 | Add_26_attn | Add | 1 × 1049028 × 4096 |
| 167 | RMSNorm_26_2 | RMSNorm | 1 × 1049028 × 4096 |
| 168 | MoE_26 | MoE Layer | 1 × 1049028 × 4096 |
| 169 | Add_26_ffn | Add | 1 × 1049028 × 4096 |
| 170 | RMSNorm_27_1 | RMSNorm | 1 × 1049028 × 4096 |
| 171 | Attention_27 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 172 | Add_27_attn | Add | 1 × 1049028 × 4096 |
| 173 | RMSNorm_27_2 | RMSNorm | 1 × 1049028 × 4096 |
| 174 | MoE_27 | MoE Layer | 1 × 1049028 × 4096 |
| 175 | Add_27_ffn | Add | 1 × 1049028 × 4096 |
| 176 | RMSNorm_28_1 | RMSNorm | 1 × 1049028 × 4096 |
| 177 | Attention_28 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 178 | Add_28_attn | Add | 1 × 1049028 × 4096 |
| 179 | RMSNorm_28_2 | RMSNorm | 1 × 1049028 × 4096 |
| 180 | MoE_28 | MoE Layer | 1 × 1049028 × 4096 |
| 181 | Add_28_ffn | Add | 1 × 1049028 × 4096 |
| 182 | RMSNorm_29_1 | RMSNorm | 1 × 1049028 × 4096 |
| 183 | Attention_29 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 184 | Add_29_attn | Add | 1 × 1049028 × 4096 |
| 185 | RMSNorm_29_2 | RMSNorm | 1 × 1049028 × 4096 |
| 186 | MoE_29 | MoE Layer | 1 × 1049028 × 4096 |
| 187 | Add_29_ffn | Add | 1 × 1049028 × 4096 |
| 188 | RMSNorm_30_1 | RMSNorm | 1 × 1049028 × 4096 |
| 189 | Attention_30 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 190 | Add_30_attn | Add | 1 × 1049028 × 4096 |
| 191 | RMSNorm_30_2 | RMSNorm | 1 × 1049028 × 4096 |
| 192 | MoE_30 | MoE Layer | 1 × 1049028 × 4096 |
| 193 | Add_30_ffn | Add | 1 × 1049028 × 4096 |
| 194 | RMSNorm_31_1 | RMSNorm | 1 × 1049028 × 4096 |
| 195 | Attention_31 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 196 | Add_31_attn | Add | 1 × 1049028 × 4096 |
| 197 | RMSNorm_31_2 | RMSNorm | 1 × 1049028 × 4096 |
| 198 | MoE_31 | MoE Layer | 1 × 1049028 × 4096 |
| 199 | Add_31_ffn | Add | 1 × 1049028 × 4096 |
| 200 | RMSNorm_32_1 | RMSNorm | 1 × 1049028 × 4096 |
| 201 | Attention_32 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 202 | Add_32_attn | Add | 1 × 1049028 × 4096 |
| 203 | RMSNorm_32_2 | RMSNorm | 1 × 1049028 × 4096 |
| 204 | MoE_32 | MoE Layer | 1 × 1049028 × 4096 |
| 205 | Add_32_ffn | Add | 1 × 1049028 × 4096 |
| 206 | RMSNorm_33_1 | RMSNorm | 1 × 1049028 × 4096 |
| 207 | Attention_33 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 208 | Add_33_attn | Add | 1 × 1049028 × 4096 |
| 209 | RMSNorm_33_2 | RMSNorm | 1 × 1049028 × 4096 |
| 210 | MoE_33 | MoE Layer | 1 × 1049028 × 4096 |
| 211 | Add_33_ffn | Add | 1 × 1049028 × 4096 |
| 212 | RMSNorm_34_1 | RMSNorm | 1 × 1049028 × 4096 |
| 213 | Attention_34 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 214 | Add_34_attn | Add | 1 × 1049028 × 4096 |
| 215 | RMSNorm_34_2 | RMSNorm | 1 × 1049028 × 4096 |
| 216 | MoE_34 | MoE Layer | 1 × 1049028 × 4096 |
| 217 | Add_34_ffn | Add | 1 × 1049028 × 4096 |
| 218 | RMSNorm_35_1 | RMSNorm | 1 × 1049028 × 4096 |
| 219 | Attention_35 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 220 | Add_35_attn | Add | 1 × 1049028 × 4096 |
| 221 | RMSNorm_35_2 | RMSNorm | 1 × 1049028 × 4096 |
| 222 | MoE_35 | MoE Layer | 1 × 1049028 × 4096 |
| 223 | Add_35_ffn | Add | 1 × 1049028 × 4096 |
| 224 | RMSNorm_36_1 | RMSNorm | 1 × 1049028 × 4096 |
| 225 | Attention_36 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 226 | Add_36_attn | Add | 1 × 1049028 × 4096 |
| 227 | RMSNorm_36_2 | RMSNorm | 1 × 1049028 × 4096 |
| 228 | MoE_36 | MoE Layer | 1 × 1049028 × 4096 |
| 229 | Add_36_ffn | Add | 1 × 1049028 × 4096 |
| 230 | RMSNorm_37_1 | RMSNorm | 1 × 1049028 × 4096 |
| 231 | Attention_37 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 232 | Add_37_attn | Add | 1 × 1049028 × 4096 |
| 233 | RMSNorm_37_2 | RMSNorm | 1 × 1049028 × 4096 |
| 234 | MoE_37 | MoE Layer | 1 × 1049028 × 4096 |
| 235 | Add_37_ffn | Add | 1 × 1049028 × 4096 |
| 236 | RMSNorm_38_1 | RMSNorm | 1 × 1049028 × 4096 |
| 237 | Attention_38 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 238 | Add_38_attn | Add | 1 × 1049028 × 4096 |
| 239 | RMSNorm_38_2 | RMSNorm | 1 × 1049028 × 4096 |
| 240 | MoE_38 | MoE Layer | 1 × 1049028 × 4096 |
| 241 | Add_38_ffn | Add | 1 × 1049028 × 4096 |
| 242 | RMSNorm_39_1 | RMSNorm | 1 × 1049028 × 4096 |
| 243 | Attention_39 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 244 | Add_39_attn | Add | 1 × 1049028 × 4096 |
| 245 | RMSNorm_39_2 | RMSNorm | 1 × 1049028 × 4096 |
| 246 | MoE_39 | MoE Layer | 1 × 1049028 × 4096 |
| 247 | Add_39_ffn | Add | 1 × 1049028 × 4096 |
| 248 | RMSNorm_40_1 | RMSNorm | 1 × 1049028 × 4096 |
| 249 | Attention_40 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 250 | Add_40_attn | Add | 1 × 1049028 × 4096 |
| 251 | RMSNorm_40_2 | RMSNorm | 1 × 1049028 × 4096 |
| 252 | MoE_40 | MoE Layer | 1 × 1049028 × 4096 |
| 253 | Add_40_ffn | Add | 1 × 1049028 × 4096 |
| 254 | RMSNorm_41_1 | RMSNorm | 1 × 1049028 × 4096 |
| 255 | Attention_41 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 256 | Add_41_attn | Add | 1 × 1049028 × 4096 |
| 257 | RMSNorm_41_2 | RMSNorm | 1 × 1049028 × 4096 |
| 258 | MoE_41 | MoE Layer | 1 × 1049028 × 4096 |
| 259 | Add_41_ffn | Add | 1 × 1049028 × 4096 |
| 260 | RMSNorm_42_1 | RMSNorm | 1 × 1049028 × 4096 |
| 261 | Attention_42 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 262 | Add_42_attn | Add | 1 × 1049028 × 4096 |
| 263 | RMSNorm_42_2 | RMSNorm | 1 × 1049028 × 4096 |
| 264 | MoE_42 | MoE Layer | 1 × 1049028 × 4096 |
| 265 | Add_42_ffn | Add | 1 × 1049028 × 4096 |
| 266 | RMSNorm_43_1 | RMSNorm | 1 × 1049028 × 4096 |
| 267 | Attention_43 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 268 | Add_43_attn | Add | 1 × 1049028 × 4096 |
| 269 | RMSNorm_43_2 | RMSNorm | 1 × 1049028 × 4096 |
| 270 | MoE_43 | MoE Layer | 1 × 1049028 × 4096 |
| 271 | Add_43_ffn | Add | 1 × 1049028 × 4096 |
| 272 | RMSNorm_44_1 | RMSNorm | 1 × 1049028 × 4096 |
| 273 | Attention_44 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 274 | Add_44_attn | Add | 1 × 1049028 × 4096 |
| 275 | RMSNorm_44_2 | RMSNorm | 1 × 1049028 × 4096 |
| 276 | MoE_44 | MoE Layer | 1 × 1049028 × 4096 |
| 277 | Add_44_ffn | Add | 1 × 1049028 × 4096 |
| 278 | RMSNorm_45_1 | RMSNorm | 1 × 1049028 × 4096 |
| 279 | Attention_45 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 280 | Add_45_attn | Add | 1 × 1049028 × 4096 |
| 281 | RMSNorm_45_2 | RMSNorm | 1 × 1049028 × 4096 |
| 282 | MoE_45 | MoE Layer | 1 × 1049028 × 4096 |
| 283 | Add_45_ffn | Add | 1 × 1049028 × 4096 |
| 284 | RMSNorm_46_1 | RMSNorm | 1 × 1049028 × 4096 |
| 285 | Attention_46 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 286 | Add_46_attn | Add | 1 × 1049028 × 4096 |
| 287 | RMSNorm_46_2 | RMSNorm | 1 × 1049028 × 4096 |
| 288 | MoE_46 | MoE Layer | 1 × 1049028 × 4096 |
| 289 | Add_46_ffn | Add | 1 × 1049028 × 4096 |
| 290 | RMSNorm_47_1 | RMSNorm | 1 × 1049028 × 4096 |
| 291 | Attention_47 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 292 | Add_47_attn | Add | 1 × 1049028 × 4096 |
| 293 | RMSNorm_47_2 | RMSNorm | 1 × 1049028 × 4096 |
| 294 | MoE_47 | MoE Layer | 1 × 1049028 × 4096 |
| 295 | Add_47_ffn | Add | 1 × 1049028 × 4096 |
| 296 | RMSNorm_48_1 | RMSNorm | 1 × 1049028 × 4096 |
| 297 | Attention_48 | Grouped Query Attn | 1 × 1049028 × 4096 |
| 298 | Add_48_attn | Add | 1 × 1049028 × 4096 |
| 299 | RMSNorm_48_2 | RMSNorm | 1 × 1049028 × 4096 |
| 300 | MoE_48 | MoE Layer | 1 × 1049028 × 4096 |
| 301 | Add_48_ffn | Add | 1 × 1049028 × 4096 |
| 302 | Final_RMSNorm | RMSNorm | 1 × 1049028 × 4096 |
| 303 | LM_Head | Linear | 1 × 1049028 × 152576 |
| 304 | Output | Output | 1 × 1049028 × 152576 |

## Do this to your own model

```
pip install neurarch-trace
neurarch-trace XiaomiMiMo/MiMo-V2.5 --plan --share
```

- Page: https://neurarch.com/m/mimo-v2-5.html
- All models: https://neurarch.com/m/index.json
- The checks: https://neurarch.com/r/index.json