# ultravox-v0_5-llama-3_2-1b

Hugging Face: `fixie-ai/ultravox-v0_5-llama-3_2-1b`
Reconstructed from `config.json` alone. No weights were read.

| | |
|---|---|
| Derived from structure | 2,838,197,504 |
| In the published checkpoint | 683,118,592 (safetensors.total, 2026-03-11) |
| Delta | +315% |
| Comparability | custom-code |
| Layers | 230 |
| Priced on | A10G (24GB) |
| Est. one run | $52.76 |

> This repository ships its own modeling code (`auto_map`, e.g. `ultravox_config.py`), so `config.json` names a class in the repo rather than an architecture `transformers` defines. The graph below is what those config keys mean under `transformers` semantics, which is not necessarily what the repo's own file builds. A gap here is a statement about what we read, not about the checkpoint.

## Structure

| # | Layer | Type | Output shape |
|---|---|---|---|
| 1 | Input | Input | 1 × 512 |
| 2 | Embedding | Embedding | 1 × 512 × 4096 |
| 3 | RoPE | RoPE | 1 × 512 × 4096 |
| 4 | Audio input | Input | 1 × 128 × 1 |
| 5 | Audio feature embed | Projection | 1 × 128 × 1280 |
| 6 | Audio_LN_1 | LayerNorm | 1 × 128 × 1280 |
| 7 | Audio_Attn_1 | Multi-Head Attention | 1 × 128 × 1280 |
| 8 | Audio_Add_1 | Add | 1 × 128 × 1280 |
| 9 | Audio_FFN_1 | Feed Forward | 1 × 128 × 1280 |
| 10 | Audio_LN_2 | LayerNorm | 1 × 128 × 1280 |
| 11 | Audio_Attn_2 | Multi-Head Attention | 1 × 128 × 1280 |
| 12 | Audio_Add_2 | Add | 1 × 128 × 1280 |
| 13 | Audio_FFN_2 | Feed Forward | 1 × 128 × 1280 |
| 14 | Audio_LN_3 | LayerNorm | 1 × 128 × 1280 |
| 15 | Audio_Attn_3 | Multi-Head Attention | 1 × 128 × 1280 |
| 16 | Audio_Add_3 | Add | 1 × 128 × 1280 |
| 17 | Audio_FFN_3 | Feed Forward | 1 × 128 × 1280 |
| 18 | Audio_LN_4 | LayerNorm | 1 × 128 × 1280 |
| 19 | Audio_Attn_4 | Multi-Head Attention | 1 × 128 × 1280 |
| 20 | Audio_Add_4 | Add | 1 × 128 × 1280 |
| 21 | Audio_FFN_4 | Feed Forward | 1 × 128 × 1280 |
| 22 | Audio_LN_5 | LayerNorm | 1 × 128 × 1280 |
| 23 | Audio_Attn_5 | Multi-Head Attention | 1 × 128 × 1280 |
| 24 | Audio_Add_5 | Add | 1 × 128 × 1280 |
| 25 | Audio_FFN_5 | Feed Forward | 1 × 128 × 1280 |
| 26 | Audio_LN_6 | LayerNorm | 1 × 128 × 1280 |
| 27 | Audio_Attn_6 | Multi-Head Attention | 1 × 128 × 1280 |
| 28 | Audio_Add_6 | Add | 1 × 128 × 1280 |
| 29 | Audio_FFN_6 | Feed Forward | 1 × 128 × 1280 |
| 30 | Audio_LN_7 | LayerNorm | 1 × 128 × 1280 |
| 31 | Audio_Attn_7 | Multi-Head Attention | 1 × 128 × 1280 |
| 32 | Audio_Add_7 | Add | 1 × 128 × 1280 |
| 33 | Audio_FFN_7 | Feed Forward | 1 × 128 × 1280 |
| 34 | Audio_LN_8 | LayerNorm | 1 × 128 × 1280 |
| 35 | Audio_Attn_8 | Multi-Head Attention | 1 × 128 × 1280 |
| 36 | Audio_Add_8 | Add | 1 × 128 × 1280 |
| 37 | Audio_FFN_8 | Feed Forward | 1 × 128 × 1280 |
| 38 | Audio_LN_9 | LayerNorm | 1 × 128 × 1280 |
| 39 | Audio_Attn_9 | Multi-Head Attention | 1 × 128 × 1280 |
| 40 | Audio_Add_9 | Add | 1 × 128 × 1280 |
| 41 | Audio_FFN_9 | Feed Forward | 1 × 128 × 1280 |
| 42 | Audio_LN_10 | LayerNorm | 1 × 128 × 1280 |
| 43 | Audio_Attn_10 | Multi-Head Attention | 1 × 128 × 1280 |
| 44 | Audio_Add_10 | Add | 1 × 128 × 1280 |
| 45 | Audio_FFN_10 | Feed Forward | 1 × 128 × 1280 |
| 46 | Audio_LN_11 | LayerNorm | 1 × 128 × 1280 |
| 47 | Audio_Attn_11 | Multi-Head Attention | 1 × 128 × 1280 |
| 48 | Audio_Add_11 | Add | 1 × 128 × 1280 |
| 49 | Audio_FFN_11 | Feed Forward | 1 × 128 × 1280 |
| 50 | Audio_LN_12 | LayerNorm | 1 × 128 × 1280 |
| 51 | Audio_Attn_12 | Multi-Head Attention | 1 × 128 × 1280 |
| 52 | Audio_Add_12 | Add | 1 × 128 × 1280 |
| 53 | Audio_FFN_12 | Feed Forward | 1 × 128 × 1280 |
| 54 | Audio_LN_13 | LayerNorm | 1 × 128 × 1280 |
| 55 | Audio_Attn_13 | Multi-Head Attention | 1 × 128 × 1280 |
| 56 | Audio_Add_13 | Add | 1 × 128 × 1280 |
| 57 | Audio_FFN_13 | Feed Forward | 1 × 128 × 1280 |
| 58 | Audio_LN_14 | LayerNorm | 1 × 128 × 1280 |
| 59 | Audio_Attn_14 | Multi-Head Attention | 1 × 128 × 1280 |
| 60 | Audio_Add_14 | Add | 1 × 128 × 1280 |
| 61 | Audio_FFN_14 | Feed Forward | 1 × 128 × 1280 |
| 62 | Audio_LN_15 | LayerNorm | 1 × 128 × 1280 |
| 63 | Audio_Attn_15 | Multi-Head Attention | 1 × 128 × 1280 |
| 64 | Audio_Add_15 | Add | 1 × 128 × 1280 |
| 65 | Audio_FFN_15 | Feed Forward | 1 × 128 × 1280 |
| 66 | Audio_LN_16 | LayerNorm | 1 × 128 × 1280 |
| 67 | Audio_Attn_16 | Multi-Head Attention | 1 × 128 × 1280 |
| 68 | Audio_Add_16 | Add | 1 × 128 × 1280 |
| 69 | Audio_FFN_16 | Feed Forward | 1 × 128 × 1280 |
| 70 | Audio_LN_17 | LayerNorm | 1 × 128 × 1280 |
| 71 | Audio_Attn_17 | Multi-Head Attention | 1 × 128 × 1280 |
| 72 | Audio_Add_17 | Add | 1 × 128 × 1280 |
| 73 | Audio_FFN_17 | Feed Forward | 1 × 128 × 1280 |
| 74 | Audio_LN_18 | LayerNorm | 1 × 128 × 1280 |
| 75 | Audio_Attn_18 | Multi-Head Attention | 1 × 128 × 1280 |
| 76 | Audio_Add_18 | Add | 1 × 128 × 1280 |
| 77 | Audio_FFN_18 | Feed Forward | 1 × 128 × 1280 |
| 78 | Audio_LN_19 | LayerNorm | 1 × 128 × 1280 |
| 79 | Audio_Attn_19 | Multi-Head Attention | 1 × 128 × 1280 |
| 80 | Audio_Add_19 | Add | 1 × 128 × 1280 |
| 81 | Audio_FFN_19 | Feed Forward | 1 × 128 × 1280 |
| 82 | Audio_LN_20 | LayerNorm | 1 × 128 × 1280 |
| 83 | Audio_Attn_20 | Multi-Head Attention | 1 × 128 × 1280 |
| 84 | Audio_Add_20 | Add | 1 × 128 × 1280 |
| 85 | Audio_FFN_20 | Feed Forward | 1 × 128 × 1280 |
| 86 | Audio_LN_21 | LayerNorm | 1 × 128 × 1280 |
| 87 | Audio_Attn_21 | Multi-Head Attention | 1 × 128 × 1280 |
| 88 | Audio_Add_21 | Add | 1 × 128 × 1280 |
| 89 | Audio_FFN_21 | Feed Forward | 1 × 128 × 1280 |
| 90 | Audio_LN_22 | LayerNorm | 1 × 128 × 1280 |
| 91 | Audio_Attn_22 | Multi-Head Attention | 1 × 128 × 1280 |
| 92 | Audio_Add_22 | Add | 1 × 128 × 1280 |
| 93 | Audio_FFN_22 | Feed Forward | 1 × 128 × 1280 |
| 94 | Audio_LN_23 | LayerNorm | 1 × 128 × 1280 |
| 95 | Audio_Attn_23 | Multi-Head Attention | 1 × 128 × 1280 |
| 96 | Audio_Add_23 | Add | 1 × 128 × 1280 |
| 97 | Audio_FFN_23 | Feed Forward | 1 × 128 × 1280 |
| 98 | Audio_LN_24 | LayerNorm | 1 × 128 × 1280 |
| 99 | Audio_Attn_24 | Multi-Head Attention | 1 × 128 × 1280 |
| 100 | Audio_Add_24 | Add | 1 × 128 × 1280 |
| 101 | Audio_FFN_24 | Feed Forward | 1 × 128 × 1280 |
| 102 | Audio_LN_25 | LayerNorm | 1 × 128 × 1280 |
| 103 | Audio_Attn_25 | Multi-Head Attention | 1 × 128 × 1280 |
| 104 | Audio_Add_25 | Add | 1 × 128 × 1280 |
| 105 | Audio_FFN_25 | Feed Forward | 1 × 128 × 1280 |
| 106 | Audio_LN_26 | LayerNorm | 1 × 128 × 1280 |
| 107 | Audio_Attn_26 | Multi-Head Attention | 1 × 128 × 1280 |
| 108 | Audio_Add_26 | Add | 1 × 128 × 1280 |
| 109 | Audio_FFN_26 | Feed Forward | 1 × 128 × 1280 |
| 110 | Audio_LN_27 | LayerNorm | 1 × 128 × 1280 |
| 111 | Audio_Attn_27 | Multi-Head Attention | 1 × 128 × 1280 |
| 112 | Audio_Add_27 | Add | 1 × 128 × 1280 |
| 113 | Audio_FFN_27 | Feed Forward | 1 × 128 × 1280 |
| 114 | Audio_LN_28 | LayerNorm | 1 × 128 × 1280 |
| 115 | Audio_Attn_28 | Multi-Head Attention | 1 × 128 × 1280 |
| 116 | Audio_Add_28 | Add | 1 × 128 × 1280 |
| 117 | Audio_FFN_28 | Feed Forward | 1 × 128 × 1280 |
| 118 | Audio_LN_29 | LayerNorm | 1 × 128 × 1280 |
| 119 | Audio_Attn_29 | Multi-Head Attention | 1 × 128 × 1280 |
| 120 | Audio_Add_29 | Add | 1 × 128 × 1280 |
| 121 | Audio_FFN_29 | Feed Forward | 1 × 128 × 1280 |
| 122 | Audio_LN_30 | LayerNorm | 1 × 128 × 1280 |
| 123 | Audio_Attn_30 | Multi-Head Attention | 1 × 128 × 1280 |
| 124 | Audio_Add_30 | Add | 1 × 128 × 1280 |
| 125 | Audio_FFN_30 | Feed Forward | 1 × 128 × 1280 |
| 126 | Audio_LN_31 | LayerNorm | 1 × 128 × 1280 |
| 127 | Audio_Attn_31 | Multi-Head Attention | 1 × 128 × 1280 |
| 128 | Audio_Add_31 | Add | 1 × 128 × 1280 |
| 129 | Audio_FFN_31 | Feed Forward | 1 × 128 × 1280 |
| 130 | Audio_LN_32 | LayerNorm | 1 × 128 × 1280 |
| 131 | Audio_Attn_32 | Multi-Head Attention | 1 × 128 × 1280 |
| 132 | Audio_Add_32 | Add | 1 × 128 × 1280 |
| 133 | Audio_FFN_32 | Feed Forward | 1 × 128 × 1280 |
| 134 | Audio projector | Projection | 1 × 128 × 4096 |
| 135 | Audio tokens | Reshape | 1 × 256 × 4096 |
| 136 | Multimodal fusion (concat tokens) | Concatenate | 1 × 768 × 4096 |
| 137 | RMSNorm_1_1 | RMSNorm | 1 × 768 × 4096 |
| 138 | Attention_1 | Grouped Query Attn | 1 × 768 × 4096 |
| 139 | Add_1_attn | Add | 1 × 768 × 4096 |
| 140 | RMSNorm_1_2 | RMSNorm | 1 × 768 × 4096 |
| 141 | FFN_1 | SwiGLU | 1 × 768 × 4096 |
| 142 | Add_1_ffn | Add | 1 × 768 × 4096 |
| 143 | RMSNorm_2_1 | RMSNorm | 1 × 768 × 4096 |
| 144 | Attention_2 | Grouped Query Attn | 1 × 768 × 4096 |
| 145 | Add_2_attn | Add | 1 × 768 × 4096 |
| 146 | RMSNorm_2_2 | RMSNorm | 1 × 768 × 4096 |
| 147 | FFN_2 | SwiGLU | 1 × 768 × 4096 |
| 148 | Add_2_ffn | Add | 1 × 768 × 4096 |
| 149 | RMSNorm_3_1 | RMSNorm | 1 × 768 × 4096 |
| 150 | Attention_3 | Grouped Query Attn | 1 × 768 × 4096 |
| 151 | Add_3_attn | Add | 1 × 768 × 4096 |
| 152 | RMSNorm_3_2 | RMSNorm | 1 × 768 × 4096 |
| 153 | FFN_3 | SwiGLU | 1 × 768 × 4096 |
| 154 | Add_3_ffn | Add | 1 × 768 × 4096 |
| 155 | RMSNorm_4_1 | RMSNorm | 1 × 768 × 4096 |
| 156 | Attention_4 | Grouped Query Attn | 1 × 768 × 4096 |
| 157 | Add_4_attn | Add | 1 × 768 × 4096 |
| 158 | RMSNorm_4_2 | RMSNorm | 1 × 768 × 4096 |
| 159 | FFN_4 | SwiGLU | 1 × 768 × 4096 |
| 160 | Add_4_ffn | Add | 1 × 768 × 4096 |
| 161 | RMSNorm_5_1 | RMSNorm | 1 × 768 × 4096 |
| 162 | Attention_5 | Grouped Query Attn | 1 × 768 × 4096 |
| 163 | Add_5_attn | Add | 1 × 768 × 4096 |
| 164 | RMSNorm_5_2 | RMSNorm | 1 × 768 × 4096 |
| 165 | FFN_5 | SwiGLU | 1 × 768 × 4096 |
| 166 | Add_5_ffn | Add | 1 × 768 × 4096 |
| 167 | RMSNorm_6_1 | RMSNorm | 1 × 768 × 4096 |
| 168 | Attention_6 | Grouped Query Attn | 1 × 768 × 4096 |
| 169 | Add_6_attn | Add | 1 × 768 × 4096 |
| 170 | RMSNorm_6_2 | RMSNorm | 1 × 768 × 4096 |
| 171 | FFN_6 | SwiGLU | 1 × 768 × 4096 |
| 172 | Add_6_ffn | Add | 1 × 768 × 4096 |
| 173 | RMSNorm_7_1 | RMSNorm | 1 × 768 × 4096 |
| 174 | Attention_7 | Grouped Query Attn | 1 × 768 × 4096 |
| 175 | Add_7_attn | Add | 1 × 768 × 4096 |
| 176 | RMSNorm_7_2 | RMSNorm | 1 × 768 × 4096 |
| 177 | FFN_7 | SwiGLU | 1 × 768 × 4096 |
| 178 | Add_7_ffn | Add | 1 × 768 × 4096 |
| 179 | RMSNorm_8_1 | RMSNorm | 1 × 768 × 4096 |
| 180 | Attention_8 | Grouped Query Attn | 1 × 768 × 4096 |
| 181 | Add_8_attn | Add | 1 × 768 × 4096 |
| 182 | RMSNorm_8_2 | RMSNorm | 1 × 768 × 4096 |
| 183 | FFN_8 | SwiGLU | 1 × 768 × 4096 |
| 184 | Add_8_ffn | Add | 1 × 768 × 4096 |
| 185 | RMSNorm_9_1 | RMSNorm | 1 × 768 × 4096 |
| 186 | Attention_9 | Grouped Query Attn | 1 × 768 × 4096 |
| 187 | Add_9_attn | Add | 1 × 768 × 4096 |
| 188 | RMSNorm_9_2 | RMSNorm | 1 × 768 × 4096 |
| 189 | FFN_9 | SwiGLU | 1 × 768 × 4096 |
| 190 | Add_9_ffn | Add | 1 × 768 × 4096 |
| 191 | RMSNorm_10_1 | RMSNorm | 1 × 768 × 4096 |
| 192 | Attention_10 | Grouped Query Attn | 1 × 768 × 4096 |
| 193 | Add_10_attn | Add | 1 × 768 × 4096 |
| 194 | RMSNorm_10_2 | RMSNorm | 1 × 768 × 4096 |
| 195 | FFN_10 | SwiGLU | 1 × 768 × 4096 |
| 196 | Add_10_ffn | Add | 1 × 768 × 4096 |
| 197 | RMSNorm_11_1 | RMSNorm | 1 × 768 × 4096 |
| 198 | Attention_11 | Grouped Query Attn | 1 × 768 × 4096 |
| 199 | Add_11_attn | Add | 1 × 768 × 4096 |
| 200 | RMSNorm_11_2 | RMSNorm | 1 × 768 × 4096 |
| 201 | FFN_11 | SwiGLU | 1 × 768 × 4096 |
| 202 | Add_11_ffn | Add | 1 × 768 × 4096 |
| 203 | RMSNorm_12_1 | RMSNorm | 1 × 768 × 4096 |
| 204 | Attention_12 | Grouped Query Attn | 1 × 768 × 4096 |
| 205 | Add_12_attn | Add | 1 × 768 × 4096 |
| 206 | RMSNorm_12_2 | RMSNorm | 1 × 768 × 4096 |
| 207 | FFN_12 | SwiGLU | 1 × 768 × 4096 |
| 208 | Add_12_ffn | Add | 1 × 768 × 4096 |
| 209 | RMSNorm_13_1 | RMSNorm | 1 × 768 × 4096 |
| 210 | Attention_13 | Grouped Query Attn | 1 × 768 × 4096 |
| 211 | Add_13_attn | Add | 1 × 768 × 4096 |
| 212 | RMSNorm_13_2 | RMSNorm | 1 × 768 × 4096 |
| 213 | FFN_13 | SwiGLU | 1 × 768 × 4096 |
| 214 | Add_13_ffn | Add | 1 × 768 × 4096 |
| 215 | RMSNorm_14_1 | RMSNorm | 1 × 768 × 4096 |
| 216 | Attention_14 | Grouped Query Attn | 1 × 768 × 4096 |
| 217 | Add_14_attn | Add | 1 × 768 × 4096 |
| 218 | RMSNorm_14_2 | RMSNorm | 1 × 768 × 4096 |
| 219 | FFN_14 | SwiGLU | 1 × 768 × 4096 |
| 220 | Add_14_ffn | Add | 1 × 768 × 4096 |
| 221 | RMSNorm_15_1 | RMSNorm | 1 × 768 × 4096 |
| 222 | Attention_15 | Grouped Query Attn | 1 × 768 × 4096 |
| 223 | Add_15_attn | Add | 1 × 768 × 4096 |
| 224 | RMSNorm_15_2 | RMSNorm | 1 × 768 × 4096 |
| 225 | FFN_15 | SwiGLU | 1 × 768 × 4096 |
| 226 | Add_15_ffn | Add | 1 × 768 × 4096 |
| 227 | RMSNorm_16_1 | RMSNorm | 1 × 768 × 4096 |
| 228 | Attention_16 | Grouped Query Attn | 1 × 768 × 4096 |
| 229 | Add_16_attn | Add | 1 × 768 × 4096 |
| 230 | RMSNorm_16_2 | RMSNorm | 1 × 768 × 4096 |
| 231 | FFN_16 | SwiGLU | 1 × 768 × 4096 |
| 232 | Add_16_ffn | Add | 1 × 768 × 4096 |
| 233 | Output | Output | 1 × 768 × 4096 |

## Do this to your own model

```
pip install neurarch-trace
neurarch-trace fixie-ai/ultravox-v0_5-llama-3_2-1b --plan --share
```

- Page: https://neurarch.com/m/ultravox-v0-5-llama-3-2-1b.html
- All models: https://neurarch.com/m/index.json
- The checks: https://neurarch.com/r/index.json