Models / voxtral_realtime
Voxtral-Mini-4B-Realtime-2602: GPU memory to fine-tune
Fine-tuning Voxtral-Mini-4B-Realtime-2602 at batch 1 and 2048 tokens takes about 85.3 GB of GPU memory for a full fine-tune (fits one H200 141 GB), and 16.1 GB with LoRA r=16 on a bf16 base (borderline on one L4 24 GB). Estimated from the model's structure, for the towers the graph reconstructs, not measured. What fits on which card.
Reconstructed from its own config.json
with no weights read. 2.2M downloads on Hugging Face.
Our count against the checkpoint
The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.
multi-tower The config declares 2 sub-models (audio_config, text_config). The published checkpoint carries all of them; the graph below carries the towers the reader reconstructs. A gap here is a statement about what we read, not about the checkpoint.
What it costs to run
Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32, 2048 tokens over 50,000 samples (assumed; no dataset attached).
Memory: what fits on which card
Largest batch per card. Training is sized at 2048 tokens per sample, serving at 4096 tokens per sequence.
| Card | Full fine-tune | LoRA r=16 | Serve bf16, 4096 tokens |
|---|---|---|---|
| T4 16 GB | does not fit at batch 1 | does not fit at batch 1 | batch 5 |
| L4 24 GB | does not fit at batch 1 | batch 1, borderline | batch 11 |
| A100 40 GB | does not fit at batch 1 | batch 3 | batch 24 |
| A100 / H100 80 GB | 2 cards at batch 1 (252 + 38 layers) | batch 8 | batch 55 |
| H200 141 GB | batch 6 | batch 15 | batch 103 |
Training: 2048 tokens per sample (2048, or the model's context when shorter; the structure table is at the config's maximum, 131072), AdamW with bf16 autocast, SDPA attention (no score matrix stored), 1.15x headroom on every card. Full fine-tune holds weights, gradients and optimizer state (1.2 x 16 bytes per parameter) plus activations. LoRA is r=16 on every linear layer but the head, base frozen in bf16 (2 bytes per parameter), adapters at full AdamW state, same activations. A two-card split is a contiguous layer cut (pipeline parallel), computed only where one card cannot hold batch 1. "Borderline" means the card holds batch 1 on the estimate but not on the conservative peak (the estimate x 1.43, at or above the measured peak on 95% of held-out runs): measure one step before reserving it. Serving: bf16 weights plus the KV cache of 4096 tokens per sequence; prefill working set, CUDA context and framework reserve not counted.
These are estimates from the graph, not measurements. Against measured runs, held out by architecture, the estimate flagged 113 of the 131 jobs that did not fit their card and said "fits" on the other 18. Every measured run.
Structure
293 nodes. Output shapes are propagated from the input shape, batch dimension excluded, at 131072 tokens: the config's maximum context, not the 2048 the memory above is sized at.
| Layer | Type | Output shape | |
|---|---|---|---|
| 1 | Input | Input | 1 × 131072 |
| 2 | Embedding | Embedding | 1 × 131072 × 3072 |
| 3 | RoPE | RoPE | 1 × 131072 × 3072 |
| 4 | Audio input | Input | 1 × 128 × 1 |
| 5 | Audio feature embed | Projection | 1 × 128 × 1280 |
| 6 | Audio_LN_1 | LayerNorm | 1 × 128 × 1280 |
| 7 | Audio_Attn_1 | Multi-Head Attention | 1 × 128 × 1280 |
| 8 | Audio_Add_1 | Add | 1 × 128 × 1280 |
| 9 | Audio_FFN_1 | Feed Forward | 1 × 128 × 1280 |
| 10 | Audio_LN_2 | LayerNorm | 1 × 128 × 1280 |
| 11 | Audio_Attn_2 | Multi-Head Attention | 1 × 128 × 1280 |
| 12 | Audio_Add_2 | Add | 1 × 128 × 1280 |
| 13 | Audio_FFN_2 | Feed Forward | 1 × 128 × 1280 |
| 14 | Audio_LN_3 | LayerNorm | 1 × 128 × 1280 |
| 15 | Audio_Attn_3 | Multi-Head Attention | 1 × 128 × 1280 |
| 16 | Audio_Add_3 | Add | 1 × 128 × 1280 |
| 17 | Audio_FFN_3 | Feed Forward | 1 × 128 × 1280 |
| 18 | Audio_LN_4 | LayerNorm | 1 × 128 × 1280 |
| 19 | Audio_Attn_4 | Multi-Head Attention | 1 × 128 × 1280 |
| 20 | Audio_Add_4 | Add | 1 × 128 × 1280 |
| 21 | Audio_FFN_4 | Feed Forward | 1 × 128 × 1280 |
| 22 | Audio_LN_5 | LayerNorm | 1 × 128 × 1280 |
| 23 | Audio_Attn_5 | Multi-Head Attention | 1 × 128 × 1280 |
| 24 | Audio_Add_5 | Add | 1 × 128 × 1280 |
| 25 | Audio_FFN_5 | Feed Forward | 1 × 128 × 1280 |
| 26 | Audio_LN_6 | LayerNorm | 1 × 128 × 1280 |
| 27 | Audio_Attn_6 | Multi-Head Attention | 1 × 128 × 1280 |
| 28 | Audio_Add_6 | Add | 1 × 128 × 1280 |
| 29 | Audio_FFN_6 | Feed Forward | 1 × 128 × 1280 |
| 30 | Audio_LN_7 | LayerNorm | 1 × 128 × 1280 |
| 31 | Audio_Attn_7 | Multi-Head Attention | 1 × 128 × 1280 |
| 32 | Audio_Add_7 | Add | 1 × 128 × 1280 |
| 33 | Audio_FFN_7 | Feed Forward | 1 × 128 × 1280 |
| 34 | Audio_LN_8 | LayerNorm | 1 × 128 × 1280 |
| 35 | Audio_Attn_8 | Multi-Head Attention | 1 × 128 × 1280 |
| 36 | Audio_Add_8 | Add | 1 × 128 × 1280 |
| 37 | Audio_FFN_8 | Feed Forward | 1 × 128 × 1280 |
| 38 | Audio_LN_9 | LayerNorm | 1 × 128 × 1280 |
| 39 | Audio_Attn_9 | Multi-Head Attention | 1 × 128 × 1280 |
| 40 | Audio_Add_9 | Add | 1 × 128 × 1280 |
| 41 | Audio_FFN_9 | Feed Forward | 1 × 128 × 1280 |
| 42 | Audio_LN_10 | LayerNorm | 1 × 128 × 1280 |
| 43 | Audio_Attn_10 | Multi-Head Attention | 1 × 128 × 1280 |
| 44 | Audio_Add_10 | Add | 1 × 128 × 1280 |
| 45 | Audio_FFN_10 | Feed Forward | 1 × 128 × 1280 |
| 46 | Audio_LN_11 | LayerNorm | 1 × 128 × 1280 |
| 47 | Audio_Attn_11 | Multi-Head Attention | 1 × 128 × 1280 |
| 48 | Audio_Add_11 | Add | 1 × 128 × 1280 |
| 49 | Audio_FFN_11 | Feed Forward | 1 × 128 × 1280 |
| 50 | Audio_LN_12 | LayerNorm | 1 × 128 × 1280 |
| 51 | Audio_Attn_12 | Multi-Head Attention | 1 × 128 × 1280 |
| 52 | Audio_Add_12 | Add | 1 × 128 × 1280 |
| 53 | Audio_FFN_12 | Feed Forward | 1 × 128 × 1280 |
| 54 | Audio_LN_13 | LayerNorm | 1 × 128 × 1280 |
| 55 | Audio_Attn_13 | Multi-Head Attention | 1 × 128 × 1280 |
| 56 | Audio_Add_13 | Add | 1 × 128 × 1280 |
| 57 | Audio_FFN_13 | Feed Forward | 1 × 128 × 1280 |
| 58 | Audio_LN_14 | LayerNorm | 1 × 128 × 1280 |
| 59 | Audio_Attn_14 | Multi-Head Attention | 1 × 128 × 1280 |
| 60 | Audio_Add_14 | Add | 1 × 128 × 1280 |
| 61 | Audio_FFN_14 | Feed Forward | 1 × 128 × 1280 |
| 62 | Audio_LN_15 | LayerNorm | 1 × 128 × 1280 |
| 63 | Audio_Attn_15 | Multi-Head Attention | 1 × 128 × 1280 |
| 64 | Audio_Add_15 | Add | 1 × 128 × 1280 |
| 65 | Audio_FFN_15 | Feed Forward | 1 × 128 × 1280 |
| 66 | Audio_LN_16 | LayerNorm | 1 × 128 × 1280 |
| 67 | Audio_Attn_16 | Multi-Head Attention | 1 × 128 × 1280 |
| 68 | Audio_Add_16 | Add | 1 × 128 × 1280 |
| 69 | Audio_FFN_16 | Feed Forward | 1 × 128 × 1280 |
| 70 | Audio_LN_17 | LayerNorm | 1 × 128 × 1280 |
| 71 | Audio_Attn_17 | Multi-Head Attention | 1 × 128 × 1280 |
| 72 | Audio_Add_17 | Add | 1 × 128 × 1280 |
| 73 | Audio_FFN_17 | Feed Forward | 1 × 128 × 1280 |
| 74 | Audio_LN_18 | LayerNorm | 1 × 128 × 1280 |
| 75 | Audio_Attn_18 | Multi-Head Attention | 1 × 128 × 1280 |
| 76 | Audio_Add_18 | Add | 1 × 128 × 1280 |
| 77 | Audio_FFN_18 | Feed Forward | 1 × 128 × 1280 |
| 78 | Audio_LN_19 | LayerNorm | 1 × 128 × 1280 |
| 79 | Audio_Attn_19 | Multi-Head Attention | 1 × 128 × 1280 |
| 80 | Audio_Add_19 | Add | 1 × 128 × 1280 |
| 81 | Audio_FFN_19 | Feed Forward | 1 × 128 × 1280 |
| 82 | Audio_LN_20 | LayerNorm | 1 × 128 × 1280 |
| 83 | Audio_Attn_20 | Multi-Head Attention | 1 × 128 × 1280 |
| 84 | Audio_Add_20 | Add | 1 × 128 × 1280 |
| 85 | Audio_FFN_20 | Feed Forward | 1 × 128 × 1280 |
| 86 | Audio_LN_21 | LayerNorm | 1 × 128 × 1280 |
| 87 | Audio_Attn_21 | Multi-Head Attention | 1 × 128 × 1280 |
| 88 | Audio_Add_21 | Add | 1 × 128 × 1280 |
| 89 | Audio_FFN_21 | Feed Forward | 1 × 128 × 1280 |
| 90 | Audio_LN_22 | LayerNorm | 1 × 128 × 1280 |
| 91 | Audio_Attn_22 | Multi-Head Attention | 1 × 128 × 1280 |
| 92 | Audio_Add_22 | Add | 1 × 128 × 1280 |
| 93 | Audio_FFN_22 | Feed Forward | 1 × 128 × 1280 |
| 94 | Audio_LN_23 | LayerNorm | 1 × 128 × 1280 |
| 95 | Audio_Attn_23 | Multi-Head Attention | 1 × 128 × 1280 |
| 96 | Audio_Add_23 | Add | 1 × 128 × 1280 |
| 97 | Audio_FFN_23 | Feed Forward | 1 × 128 × 1280 |
| 98 | Audio_LN_24 | LayerNorm | 1 × 128 × 1280 |
| 99 | Audio_Attn_24 | Multi-Head Attention | 1 × 128 × 1280 |
| 100 | Audio_Add_24 | Add | 1 × 128 × 1280 |
| 101 | Audio_FFN_24 | Feed Forward | 1 × 128 × 1280 |
| 102 | Audio_LN_25 | LayerNorm | 1 × 128 × 1280 |
| 103 | Audio_Attn_25 | Multi-Head Attention | 1 × 128 × 1280 |
| 104 | Audio_Add_25 | Add | 1 × 128 × 1280 |
| 105 | Audio_FFN_25 | Feed Forward | 1 × 128 × 1280 |
| 106 | Audio_LN_26 | LayerNorm | 1 × 128 × 1280 |
| 107 | Audio_Attn_26 | Multi-Head Attention | 1 × 128 × 1280 |
| 108 | Audio_Add_26 | Add | 1 × 128 × 1280 |
| 109 | Audio_FFN_26 | Feed Forward | 1 × 128 × 1280 |
| 110 | Audio_LN_27 | LayerNorm | 1 × 128 × 1280 |
| 111 | Audio_Attn_27 | Multi-Head Attention | 1 × 128 × 1280 |
| 112 | Audio_Add_27 | Add | 1 × 128 × 1280 |
| 113 | Audio_FFN_27 | Feed Forward | 1 × 128 × 1280 |
| 114 | Audio_LN_28 | LayerNorm | 1 × 128 × 1280 |
| 115 | Audio_Attn_28 | Multi-Head Attention | 1 × 128 × 1280 |
| 116 | Audio_Add_28 | Add | 1 × 128 × 1280 |
| 117 | Audio_FFN_28 | Feed Forward | 1 × 128 × 1280 |
| 118 | Audio_LN_29 | LayerNorm | 1 × 128 × 1280 |
| 119 | Audio_Attn_29 | Multi-Head Attention | 1 × 128 × 1280 |
| 120 | Audio_Add_29 | Add | 1 × 128 × 1280 |
| 121 | Audio_FFN_29 | Feed Forward | 1 × 128 × 1280 |
| 122 | Audio_LN_30 | LayerNorm | 1 × 128 × 1280 |
| 123 | Audio_Attn_30 | Multi-Head Attention | 1 × 128 × 1280 |
| 124 | Audio_Add_30 | Add | 1 × 128 × 1280 |
| 125 | Audio_FFN_30 | Feed Forward | 1 × 128 × 1280 |
| 126 | Audio_LN_31 | LayerNorm | 1 × 128 × 1280 |
| 127 | Audio_Attn_31 | Multi-Head Attention | 1 × 128 × 1280 |
| 128 | Audio_Add_31 | Add | 1 × 128 × 1280 |
| 129 | Audio_FFN_31 | Feed Forward | 1 × 128 × 1280 |
| 130 | Audio_LN_32 | LayerNorm | 1 × 128 × 1280 |
| 131 | Audio_Attn_32 | Multi-Head Attention | 1 × 128 × 1280 |
| 132 | Audio_Add_32 | Add | 1 × 128 × 1280 |
| 133 | Audio_FFN_32 | Feed Forward | 1 × 128 × 1280 |
| 134 | Audio projector | Projection | 1 × 128 × 3072 |
| 135 | Audio tokens | Reshape | 1 × 256 × 3072 |
| 136 | Multimodal fusion (concat tokens) | Concatenate | 1 × 131328 × 3072 |
| 137 | RMSNorm_1_1 | RMSNorm | 1 × 131328 × 3072 |
| 138 | Attention_1 | Grouped Query Attn | 1 × 131328 × 3072 |
| 139 | Add_1_attn | Add | 1 × 131328 × 3072 |
| 140 | RMSNorm_1_2 | RMSNorm | 1 × 131328 × 3072 |
| 141 | FFN_1 | SwiGLU | 1 × 131328 × 3072 |
| 142 | Add_1_ffn | Add | 1 × 131328 × 3072 |
| 143 | RMSNorm_2_1 | RMSNorm | 1 × 131328 × 3072 |
| 144 | Attention_2 | Grouped Query Attn | 1 × 131328 × 3072 |
| 145 | Add_2_attn | Add | 1 × 131328 × 3072 |
| 146 | RMSNorm_2_2 | RMSNorm | 1 × 131328 × 3072 |
| 147 | FFN_2 | SwiGLU | 1 × 131328 × 3072 |
| 148 | Add_2_ffn | Add | 1 × 131328 × 3072 |
| 149 | RMSNorm_3_1 | RMSNorm | 1 × 131328 × 3072 |
| 150 | Attention_3 | Grouped Query Attn | 1 × 131328 × 3072 |
| 151 | Add_3_attn | Add | 1 × 131328 × 3072 |
| 152 | RMSNorm_3_2 | RMSNorm | 1 × 131328 × 3072 |
| 153 | FFN_3 | SwiGLU | 1 × 131328 × 3072 |
| 154 | Add_3_ffn | Add | 1 × 131328 × 3072 |
| 155 | RMSNorm_4_1 | RMSNorm | 1 × 131328 × 3072 |
| 156 | Attention_4 | Grouped Query Attn | 1 × 131328 × 3072 |
| 157 | Add_4_attn | Add | 1 × 131328 × 3072 |
| 158 | RMSNorm_4_2 | RMSNorm | 1 × 131328 × 3072 |
| 159 | FFN_4 | SwiGLU | 1 × 131328 × 3072 |
| 160 | Add_4_ffn | Add | 1 × 131328 × 3072 |
| 161 | RMSNorm_5_1 | RMSNorm | 1 × 131328 × 3072 |
| 162 | Attention_5 | Grouped Query Attn | 1 × 131328 × 3072 |
| 163 | Add_5_attn | Add | 1 × 131328 × 3072 |
| 164 | RMSNorm_5_2 | RMSNorm | 1 × 131328 × 3072 |
| 165 | FFN_5 | SwiGLU | 1 × 131328 × 3072 |
| 166 | Add_5_ffn | Add | 1 × 131328 × 3072 |
| 167 | RMSNorm_6_1 | RMSNorm | 1 × 131328 × 3072 |
| 168 | Attention_6 | Grouped Query Attn | 1 × 131328 × 3072 |
| 169 | Add_6_attn | Add | 1 × 131328 × 3072 |
| 170 | RMSNorm_6_2 | RMSNorm | 1 × 131328 × 3072 |
| 171 | FFN_6 | SwiGLU | 1 × 131328 × 3072 |
| 172 | Add_6_ffn | Add | 1 × 131328 × 3072 |
| 173 | RMSNorm_7_1 | RMSNorm | 1 × 131328 × 3072 |
| 174 | Attention_7 | Grouped Query Attn | 1 × 131328 × 3072 |
| 175 | Add_7_attn | Add | 1 × 131328 × 3072 |
| 176 | RMSNorm_7_2 | RMSNorm | 1 × 131328 × 3072 |
| 177 | FFN_7 | SwiGLU | 1 × 131328 × 3072 |
| 178 | Add_7_ffn | Add | 1 × 131328 × 3072 |
| 179 | RMSNorm_8_1 | RMSNorm | 1 × 131328 × 3072 |
| 180 | Attention_8 | Grouped Query Attn | 1 × 131328 × 3072 |
| 181 | Add_8_attn | Add | 1 × 131328 × 3072 |
| 182 | RMSNorm_8_2 | RMSNorm | 1 × 131328 × 3072 |
| 183 | FFN_8 | SwiGLU | 1 × 131328 × 3072 |
| 184 | Add_8_ffn | Add | 1 × 131328 × 3072 |
| 185 | RMSNorm_9_1 | RMSNorm | 1 × 131328 × 3072 |
| 186 | Attention_9 | Grouped Query Attn | 1 × 131328 × 3072 |
| 187 | Add_9_attn | Add | 1 × 131328 × 3072 |
| 188 | RMSNorm_9_2 | RMSNorm | 1 × 131328 × 3072 |
| 189 | FFN_9 | SwiGLU | 1 × 131328 × 3072 |
| 190 | Add_9_ffn | Add | 1 × 131328 × 3072 |
| 191 | RMSNorm_10_1 | RMSNorm | 1 × 131328 × 3072 |
| 192 | Attention_10 | Grouped Query Attn | 1 × 131328 × 3072 |
| 193 | Add_10_attn | Add | 1 × 131328 × 3072 |
| 194 | RMSNorm_10_2 | RMSNorm | 1 × 131328 × 3072 |
| 195 | FFN_10 | SwiGLU | 1 × 131328 × 3072 |
| 196 | Add_10_ffn | Add | 1 × 131328 × 3072 |
| 197 | RMSNorm_11_1 | RMSNorm | 1 × 131328 × 3072 |
| 198 | Attention_11 | Grouped Query Attn | 1 × 131328 × 3072 |
| 199 | Add_11_attn | Add | 1 × 131328 × 3072 |
| 200 | RMSNorm_11_2 | RMSNorm | 1 × 131328 × 3072 |
| 201 | FFN_11 | SwiGLU | 1 × 131328 × 3072 |
| 202 | Add_11_ffn | Add | 1 × 131328 × 3072 |
| 203 | RMSNorm_12_1 | RMSNorm | 1 × 131328 × 3072 |
| 204 | Attention_12 | Grouped Query Attn | 1 × 131328 × 3072 |
| 205 | Add_12_attn | Add | 1 × 131328 × 3072 |
| 206 | RMSNorm_12_2 | RMSNorm | 1 × 131328 × 3072 |
| 207 | FFN_12 | SwiGLU | 1 × 131328 × 3072 |
| 208 | Add_12_ffn | Add | 1 × 131328 × 3072 |
| 209 | RMSNorm_13_1 | RMSNorm | 1 × 131328 × 3072 |
| 210 | Attention_13 | Grouped Query Attn | 1 × 131328 × 3072 |
| 211 | Add_13_attn | Add | 1 × 131328 × 3072 |
| 212 | RMSNorm_13_2 | RMSNorm | 1 × 131328 × 3072 |
| 213 | FFN_13 | SwiGLU | 1 × 131328 × 3072 |
| 214 | Add_13_ffn | Add | 1 × 131328 × 3072 |
| 215 | RMSNorm_14_1 | RMSNorm | 1 × 131328 × 3072 |
| 216 | Attention_14 | Grouped Query Attn | 1 × 131328 × 3072 |
| 217 | Add_14_attn | Add | 1 × 131328 × 3072 |
| 218 | RMSNorm_14_2 | RMSNorm | 1 × 131328 × 3072 |
| 219 | FFN_14 | SwiGLU | 1 × 131328 × 3072 |
| 220 | Add_14_ffn | Add | 1 × 131328 × 3072 |
| 221 | RMSNorm_15_1 | RMSNorm | 1 × 131328 × 3072 |
| 222 | Attention_15 | Grouped Query Attn | 1 × 131328 × 3072 |
| 223 | Add_15_attn | Add | 1 × 131328 × 3072 |
| 224 | RMSNorm_15_2 | RMSNorm | 1 × 131328 × 3072 |
| 225 | FFN_15 | SwiGLU | 1 × 131328 × 3072 |
| 226 | Add_15_ffn | Add | 1 × 131328 × 3072 |
| 227 | RMSNorm_16_1 | RMSNorm | 1 × 131328 × 3072 |
| 228 | Attention_16 | Grouped Query Attn | 1 × 131328 × 3072 |
| 229 | Add_16_attn | Add | 1 × 131328 × 3072 |
| 230 | RMSNorm_16_2 | RMSNorm | 1 × 131328 × 3072 |
| 231 | FFN_16 | SwiGLU | 1 × 131328 × 3072 |
| 232 | Add_16_ffn | Add | 1 × 131328 × 3072 |
| 233 | RMSNorm_17_1 | RMSNorm | 1 × 131328 × 3072 |
| 234 | Attention_17 | Grouped Query Attn | 1 × 131328 × 3072 |
| 235 | Add_17_attn | Add | 1 × 131328 × 3072 |
| 236 | RMSNorm_17_2 | RMSNorm | 1 × 131328 × 3072 |
| 237 | FFN_17 | SwiGLU | 1 × 131328 × 3072 |
| 238 | Add_17_ffn | Add | 1 × 131328 × 3072 |
| 239 | RMSNorm_18_1 | RMSNorm | 1 × 131328 × 3072 |
| 240 | Attention_18 | Grouped Query Attn | 1 × 131328 × 3072 |
| 241 | Add_18_attn | Add | 1 × 131328 × 3072 |
| 242 | RMSNorm_18_2 | RMSNorm | 1 × 131328 × 3072 |
| 243 | FFN_18 | SwiGLU | 1 × 131328 × 3072 |
| 244 | Add_18_ffn | Add | 1 × 131328 × 3072 |
| 245 | RMSNorm_19_1 | RMSNorm | 1 × 131328 × 3072 |
| 246 | Attention_19 | Grouped Query Attn | 1 × 131328 × 3072 |
| 247 | Add_19_attn | Add | 1 × 131328 × 3072 |
| 248 | RMSNorm_19_2 | RMSNorm | 1 × 131328 × 3072 |
| 249 | FFN_19 | SwiGLU | 1 × 131328 × 3072 |
| 250 | Add_19_ffn | Add | 1 × 131328 × 3072 |
| 251 | RMSNorm_20_1 | RMSNorm | 1 × 131328 × 3072 |
| 252 | Attention_20 | Grouped Query Attn | 1 × 131328 × 3072 |
| 253 | Add_20_attn | Add | 1 × 131328 × 3072 |
| 254 | RMSNorm_20_2 | RMSNorm | 1 × 131328 × 3072 |
| 255 | FFN_20 | SwiGLU | 1 × 131328 × 3072 |
| 256 | Add_20_ffn | Add | 1 × 131328 × 3072 |
| 257 | RMSNorm_21_1 | RMSNorm | 1 × 131328 × 3072 |
| 258 | Attention_21 | Grouped Query Attn | 1 × 131328 × 3072 |
| 259 | Add_21_attn | Add | 1 × 131328 × 3072 |
| 260 | RMSNorm_21_2 | RMSNorm | 1 × 131328 × 3072 |
| 261 | FFN_21 | SwiGLU | 1 × 131328 × 3072 |
| 262 | Add_21_ffn | Add | 1 × 131328 × 3072 |
| 263 | RMSNorm_22_1 | RMSNorm | 1 × 131328 × 3072 |
| 264 | Attention_22 | Grouped Query Attn | 1 × 131328 × 3072 |
| 265 | Add_22_attn | Add | 1 × 131328 × 3072 |
| 266 | RMSNorm_22_2 | RMSNorm | 1 × 131328 × 3072 |
| 267 | FFN_22 | SwiGLU | 1 × 131328 × 3072 |
| 268 | Add_22_ffn | Add | 1 × 131328 × 3072 |
| 269 | RMSNorm_23_1 | RMSNorm | 1 × 131328 × 3072 |
| 270 | Attention_23 | Grouped Query Attn | 1 × 131328 × 3072 |
| 271 | Add_23_attn | Add | 1 × 131328 × 3072 |
| 272 | RMSNorm_23_2 | RMSNorm | 1 × 131328 × 3072 |
| 273 | FFN_23 | SwiGLU | 1 × 131328 × 3072 |
| 274 | Add_23_ffn | Add | 1 × 131328 × 3072 |
| 275 | RMSNorm_24_1 | RMSNorm | 1 × 131328 × 3072 |
| 276 | Attention_24 | Grouped Query Attn | 1 × 131328 × 3072 |
| 277 | Add_24_attn | Add | 1 × 131328 × 3072 |
| 278 | RMSNorm_24_2 | RMSNorm | 1 × 131328 × 3072 |
| 279 | FFN_24 | SwiGLU | 1 × 131328 × 3072 |
| 280 | Add_24_ffn | Add | 1 × 131328 × 3072 |
| 281 | RMSNorm_25_1 | RMSNorm | 1 × 131328 × 3072 |
| 282 | Attention_25 | Grouped Query Attn | 1 × 131328 × 3072 |
| 283 | Add_25_attn | Add | 1 × 131328 × 3072 |
| 284 | RMSNorm_25_2 | RMSNorm | 1 × 131328 × 3072 |
| 285 | FFN_25 | SwiGLU | 1 × 131328 × 3072 |
| 286 | Add_25_ffn | Add | 1 × 131328 × 3072 |
| 287 | RMSNorm_26_1 | RMSNorm | 1 × 131328 × 3072 |
| 288 | Attention_26 | Grouped Query Attn | 1 × 131328 × 3072 |
| 289 | Add_26_attn | Add | 1 × 131328 × 3072 |
| 290 | RMSNorm_26_2 | RMSNorm | 1 × 131328 × 3072 |
| 291 | FFN_26 | SwiGLU | 1 × 131328 × 3072 |
| 292 | Add_26_ffn | Add | 1 × 131328 × 3072 |
| 293 | Output | Output | 1 × 131328 × 3072 |
What the verifier says
dead-end
deep-attention-default-init
What changed upstream
The change tracker follows mistralai/Voxtral-Mini-4B-Realtime-2602: every revision it has read, what changed in the config, tokenizer, chat template or weights between them, and an OSV advisory per change.
Do this to your own model
Same numbers, on a model in your repo, in one command. No account.
pip install neurarch-trace
neurarch-trace mistralai/Voxtral-Mini-4B-Realtime-2602 --plan --share