# Llama-3_3-Nemotron-Super-49B-v1_5-FP8

Hugging Face: `nvidia/Llama-3_3-Nemotron-Super-49B-v1_5-FP8`
Reconstructed from `config.json` alone. No weights were read.

| | |
|---|---|
| Derived from structure | 29,617,427,712 |
| In the published checkpoint | 49,867,145,216 (safetensors.total, 2025-10-15) |
| Delta | -40.6% |
| Comparability | custom-code |
| Layers | 484 |
| Priced on | A10G (24GB) |
| Est. one run | $561534.71 |

> This repository ships its own modeling code (`auto_map`, e.g. `configuration_decilm.py`), so `config.json` names a class in the repo rather than an architecture `transformers` defines. The graph below is what those config keys mean under `transformers` semantics, which is not necessarily what the repo's own file builds. A gap here is a statement about what we read, not about the checkpoint.

## Structure

| # | Layer | Type | Output shape |
|---|---|---|---|
| 1 | Input | Input | 1 × 131072 |
| 2 | Embedding | Embedding | 1 × 131072 × 8192 |
| 3 | RoPE | RoPE | 1 × 131072 × 8192 |
| 4 | RMSNorm_1_1 | RMSNorm | 1 × 131072 × 8192 |
| 5 | Attention_1 | Grouped Query Attn | 1 × 131072 × 8192 |
| 6 | Add_1_attn | Add | 1 × 131072 × 8192 |
| 7 | RMSNorm_1_2 | RMSNorm | 1 × 131072 × 8192 |
| 8 | FFN_1 | SwiGLU | 1 × 131072 × 8192 |
| 9 | Add_1_ffn | Add | 1 × 131072 × 8192 |
| 10 | RMSNorm_2_1 | RMSNorm | 1 × 131072 × 8192 |
| 11 | Attention_2 | Grouped Query Attn | 1 × 131072 × 8192 |
| 12 | Add_2_attn | Add | 1 × 131072 × 8192 |
| 13 | RMSNorm_2_2 | RMSNorm | 1 × 131072 × 8192 |
| 14 | FFN_2 | SwiGLU | 1 × 131072 × 8192 |
| 15 | Add_2_ffn | Add | 1 × 131072 × 8192 |
| 16 | RMSNorm_3_1 | RMSNorm | 1 × 131072 × 8192 |
| 17 | Attention_3 | Grouped Query Attn | 1 × 131072 × 8192 |
| 18 | Add_3_attn | Add | 1 × 131072 × 8192 |
| 19 | RMSNorm_3_2 | RMSNorm | 1 × 131072 × 8192 |
| 20 | FFN_3 | SwiGLU | 1 × 131072 × 8192 |
| 21 | Add_3_ffn | Add | 1 × 131072 × 8192 |
| 22 | RMSNorm_4_1 | RMSNorm | 1 × 131072 × 8192 |
| 23 | Attention_4 | Grouped Query Attn | 1 × 131072 × 8192 |
| 24 | Add_4_attn | Add | 1 × 131072 × 8192 |
| 25 | RMSNorm_4_2 | RMSNorm | 1 × 131072 × 8192 |
| 26 | FFN_4 | SwiGLU | 1 × 131072 × 8192 |
| 27 | Add_4_ffn | Add | 1 × 131072 × 8192 |
| 28 | RMSNorm_5_1 | RMSNorm | 1 × 131072 × 8192 |
| 29 | Attention_5 | Grouped Query Attn | 1 × 131072 × 8192 |
| 30 | Add_5_attn | Add | 1 × 131072 × 8192 |
| 31 | RMSNorm_5_2 | RMSNorm | 1 × 131072 × 8192 |
| 32 | FFN_5 | SwiGLU | 1 × 131072 × 8192 |
| 33 | Add_5_ffn | Add | 1 × 131072 × 8192 |
| 34 | RMSNorm_6_1 | RMSNorm | 1 × 131072 × 8192 |
| 35 | Attention_6 | Grouped Query Attn | 1 × 131072 × 8192 |
| 36 | Add_6_attn | Add | 1 × 131072 × 8192 |
| 37 | RMSNorm_6_2 | RMSNorm | 1 × 131072 × 8192 |
| 38 | FFN_6 | SwiGLU | 1 × 131072 × 8192 |
| 39 | Add_6_ffn | Add | 1 × 131072 × 8192 |
| 40 | RMSNorm_7_1 | RMSNorm | 1 × 131072 × 8192 |
| 41 | Attention_7 | Grouped Query Attn | 1 × 131072 × 8192 |
| 42 | Add_7_attn | Add | 1 × 131072 × 8192 |
| 43 | RMSNorm_7_2 | RMSNorm | 1 × 131072 × 8192 |
| 44 | FFN_7 | SwiGLU | 1 × 131072 × 8192 |
| 45 | Add_7_ffn | Add | 1 × 131072 × 8192 |
| 46 | RMSNorm_8_1 | RMSNorm | 1 × 131072 × 8192 |
| 47 | Attention_8 | Grouped Query Attn | 1 × 131072 × 8192 |
| 48 | Add_8_attn | Add | 1 × 131072 × 8192 |
| 49 | RMSNorm_8_2 | RMSNorm | 1 × 131072 × 8192 |
| 50 | FFN_8 | SwiGLU | 1 × 131072 × 8192 |
| 51 | Add_8_ffn | Add | 1 × 131072 × 8192 |
| 52 | RMSNorm_9_1 | RMSNorm | 1 × 131072 × 8192 |
| 53 | Attention_9 | Grouped Query Attn | 1 × 131072 × 8192 |
| 54 | Add_9_attn | Add | 1 × 131072 × 8192 |
| 55 | RMSNorm_9_2 | RMSNorm | 1 × 131072 × 8192 |
| 56 | FFN_9 | SwiGLU | 1 × 131072 × 8192 |
| 57 | Add_9_ffn | Add | 1 × 131072 × 8192 |
| 58 | RMSNorm_10_1 | RMSNorm | 1 × 131072 × 8192 |
| 59 | Attention_10 | Grouped Query Attn | 1 × 131072 × 8192 |
| 60 | Add_10_attn | Add | 1 × 131072 × 8192 |
| 61 | RMSNorm_10_2 | RMSNorm | 1 × 131072 × 8192 |
| 62 | FFN_10 | SwiGLU | 1 × 131072 × 8192 |
| 63 | Add_10_ffn | Add | 1 × 131072 × 8192 |
| 64 | RMSNorm_11_1 | RMSNorm | 1 × 131072 × 8192 |
| 65 | Attention_11 | Grouped Query Attn | 1 × 131072 × 8192 |
| 66 | Add_11_attn | Add | 1 × 131072 × 8192 |
| 67 | RMSNorm_11_2 | RMSNorm | 1 × 131072 × 8192 |
| 68 | FFN_11 | SwiGLU | 1 × 131072 × 8192 |
| 69 | Add_11_ffn | Add | 1 × 131072 × 8192 |
| 70 | RMSNorm_12_1 | RMSNorm | 1 × 131072 × 8192 |
| 71 | Attention_12 | Grouped Query Attn | 1 × 131072 × 8192 |
| 72 | Add_12_attn | Add | 1 × 131072 × 8192 |
| 73 | RMSNorm_12_2 | RMSNorm | 1 × 131072 × 8192 |
| 74 | FFN_12 | SwiGLU | 1 × 131072 × 8192 |
| 75 | Add_12_ffn | Add | 1 × 131072 × 8192 |
| 76 | RMSNorm_13_1 | RMSNorm | 1 × 131072 × 8192 |
| 77 | Attention_13 | Grouped Query Attn | 1 × 131072 × 8192 |
| 78 | Add_13_attn | Add | 1 × 131072 × 8192 |
| 79 | RMSNorm_13_2 | RMSNorm | 1 × 131072 × 8192 |
| 80 | FFN_13 | SwiGLU | 1 × 131072 × 8192 |
| 81 | Add_13_ffn | Add | 1 × 131072 × 8192 |
| 82 | RMSNorm_14_1 | RMSNorm | 1 × 131072 × 8192 |
| 83 | Attention_14 | Grouped Query Attn | 1 × 131072 × 8192 |
| 84 | Add_14_attn | Add | 1 × 131072 × 8192 |
| 85 | RMSNorm_14_2 | RMSNorm | 1 × 131072 × 8192 |
| 86 | FFN_14 | SwiGLU | 1 × 131072 × 8192 |
| 87 | Add_14_ffn | Add | 1 × 131072 × 8192 |
| 88 | RMSNorm_15_1 | RMSNorm | 1 × 131072 × 8192 |
| 89 | Attention_15 | Grouped Query Attn | 1 × 131072 × 8192 |
| 90 | Add_15_attn | Add | 1 × 131072 × 8192 |
| 91 | RMSNorm_15_2 | RMSNorm | 1 × 131072 × 8192 |
| 92 | FFN_15 | SwiGLU | 1 × 131072 × 8192 |
| 93 | Add_15_ffn | Add | 1 × 131072 × 8192 |
| 94 | RMSNorm_16_1 | RMSNorm | 1 × 131072 × 8192 |
| 95 | Attention_16 | Grouped Query Attn | 1 × 131072 × 8192 |
| 96 | Add_16_attn | Add | 1 × 131072 × 8192 |
| 97 | RMSNorm_16_2 | RMSNorm | 1 × 131072 × 8192 |
| 98 | FFN_16 | SwiGLU | 1 × 131072 × 8192 |
| 99 | Add_16_ffn | Add | 1 × 131072 × 8192 |
| 100 | RMSNorm_17_1 | RMSNorm | 1 × 131072 × 8192 |
| 101 | Attention_17 | Grouped Query Attn | 1 × 131072 × 8192 |
| 102 | Add_17_attn | Add | 1 × 131072 × 8192 |
| 103 | RMSNorm_17_2 | RMSNorm | 1 × 131072 × 8192 |
| 104 | FFN_17 | SwiGLU | 1 × 131072 × 8192 |
| 105 | Add_17_ffn | Add | 1 × 131072 × 8192 |
| 106 | RMSNorm_18_1 | RMSNorm | 1 × 131072 × 8192 |
| 107 | Attention_18 | Grouped Query Attn | 1 × 131072 × 8192 |
| 108 | Add_18_attn | Add | 1 × 131072 × 8192 |
| 109 | RMSNorm_18_2 | RMSNorm | 1 × 131072 × 8192 |
| 110 | FFN_18 | SwiGLU | 1 × 131072 × 8192 |
| 111 | Add_18_ffn | Add | 1 × 131072 × 8192 |
| 112 | RMSNorm_19_1 | RMSNorm | 1 × 131072 × 8192 |
| 113 | Attention_19 | Grouped Query Attn | 1 × 131072 × 8192 |
| 114 | Add_19_attn | Add | 1 × 131072 × 8192 |
| 115 | RMSNorm_19_2 | RMSNorm | 1 × 131072 × 8192 |
| 116 | FFN_19 | SwiGLU | 1 × 131072 × 8192 |
| 117 | Add_19_ffn | Add | 1 × 131072 × 8192 |
| 118 | RMSNorm_20_1 | RMSNorm | 1 × 131072 × 8192 |
| 119 | Attention_20 | Grouped Query Attn | 1 × 131072 × 8192 |
| 120 | Add_20_attn | Add | 1 × 131072 × 8192 |
| 121 | RMSNorm_20_2 | RMSNorm | 1 × 131072 × 8192 |
| 122 | FFN_20 | SwiGLU | 1 × 131072 × 8192 |
| 123 | Add_20_ffn | Add | 1 × 131072 × 8192 |
| 124 | RMSNorm_21_1 | RMSNorm | 1 × 131072 × 8192 |
| 125 | Attention_21 | Grouped Query Attn | 1 × 131072 × 8192 |
| 126 | Add_21_attn | Add | 1 × 131072 × 8192 |
| 127 | RMSNorm_21_2 | RMSNorm | 1 × 131072 × 8192 |
| 128 | FFN_21 | SwiGLU | 1 × 131072 × 8192 |
| 129 | Add_21_ffn | Add | 1 × 131072 × 8192 |
| 130 | RMSNorm_22_1 | RMSNorm | 1 × 131072 × 8192 |
| 131 | Attention_22 | Grouped Query Attn | 1 × 131072 × 8192 |
| 132 | Add_22_attn | Add | 1 × 131072 × 8192 |
| 133 | RMSNorm_22_2 | RMSNorm | 1 × 131072 × 8192 |
| 134 | FFN_22 | SwiGLU | 1 × 131072 × 8192 |
| 135 | Add_22_ffn | Add | 1 × 131072 × 8192 |
| 136 | RMSNorm_23_1 | RMSNorm | 1 × 131072 × 8192 |
| 137 | Attention_23 | Grouped Query Attn | 1 × 131072 × 8192 |
| 138 | Add_23_attn | Add | 1 × 131072 × 8192 |
| 139 | RMSNorm_23_2 | RMSNorm | 1 × 131072 × 8192 |
| 140 | FFN_23 | SwiGLU | 1 × 131072 × 8192 |
| 141 | Add_23_ffn | Add | 1 × 131072 × 8192 |
| 142 | RMSNorm_24_1 | RMSNorm | 1 × 131072 × 8192 |
| 143 | Attention_24 | Grouped Query Attn | 1 × 131072 × 8192 |
| 144 | Add_24_attn | Add | 1 × 131072 × 8192 |
| 145 | RMSNorm_24_2 | RMSNorm | 1 × 131072 × 8192 |
| 146 | FFN_24 | SwiGLU | 1 × 131072 × 8192 |
| 147 | Add_24_ffn | Add | 1 × 131072 × 8192 |
| 148 | RMSNorm_25_1 | RMSNorm | 1 × 131072 × 8192 |
| 149 | Attention_25 | Grouped Query Attn | 1 × 131072 × 8192 |
| 150 | Add_25_attn | Add | 1 × 131072 × 8192 |
| 151 | RMSNorm_25_2 | RMSNorm | 1 × 131072 × 8192 |
| 152 | FFN_25 | SwiGLU | 1 × 131072 × 8192 |
| 153 | Add_25_ffn | Add | 1 × 131072 × 8192 |
| 154 | RMSNorm_26_1 | RMSNorm | 1 × 131072 × 8192 |
| 155 | Attention_26 | Grouped Query Attn | 1 × 131072 × 8192 |
| 156 | Add_26_attn | Add | 1 × 131072 × 8192 |
| 157 | RMSNorm_26_2 | RMSNorm | 1 × 131072 × 8192 |
| 158 | FFN_26 | SwiGLU | 1 × 131072 × 8192 |
| 159 | Add_26_ffn | Add | 1 × 131072 × 8192 |
| 160 | RMSNorm_27_1 | RMSNorm | 1 × 131072 × 8192 |
| 161 | Attention_27 | Grouped Query Attn | 1 × 131072 × 8192 |
| 162 | Add_27_attn | Add | 1 × 131072 × 8192 |
| 163 | RMSNorm_27_2 | RMSNorm | 1 × 131072 × 8192 |
| 164 | FFN_27 | SwiGLU | 1 × 131072 × 8192 |
| 165 | Add_27_ffn | Add | 1 × 131072 × 8192 |
| 166 | RMSNorm_28_1 | RMSNorm | 1 × 131072 × 8192 |
| 167 | Attention_28 | Grouped Query Attn | 1 × 131072 × 8192 |
| 168 | Add_28_attn | Add | 1 × 131072 × 8192 |
| 169 | RMSNorm_28_2 | RMSNorm | 1 × 131072 × 8192 |
| 170 | FFN_28 | SwiGLU | 1 × 131072 × 8192 |
| 171 | Add_28_ffn | Add | 1 × 131072 × 8192 |
| 172 | RMSNorm_29_1 | RMSNorm | 1 × 131072 × 8192 |
| 173 | Attention_29 | Grouped Query Attn | 1 × 131072 × 8192 |
| 174 | Add_29_attn | Add | 1 × 131072 × 8192 |
| 175 | RMSNorm_29_2 | RMSNorm | 1 × 131072 × 8192 |
| 176 | FFN_29 | SwiGLU | 1 × 131072 × 8192 |
| 177 | Add_29_ffn | Add | 1 × 131072 × 8192 |
| 178 | RMSNorm_30_1 | RMSNorm | 1 × 131072 × 8192 |
| 179 | Attention_30 | Grouped Query Attn | 1 × 131072 × 8192 |
| 180 | Add_30_attn | Add | 1 × 131072 × 8192 |
| 181 | RMSNorm_30_2 | RMSNorm | 1 × 131072 × 8192 |
| 182 | FFN_30 | SwiGLU | 1 × 131072 × 8192 |
| 183 | Add_30_ffn | Add | 1 × 131072 × 8192 |
| 184 | RMSNorm_31_1 | RMSNorm | 1 × 131072 × 8192 |
| 185 | Attention_31 | Grouped Query Attn | 1 × 131072 × 8192 |
| 186 | Add_31_attn | Add | 1 × 131072 × 8192 |
| 187 | RMSNorm_31_2 | RMSNorm | 1 × 131072 × 8192 |
| 188 | FFN_31 | SwiGLU | 1 × 131072 × 8192 |
| 189 | Add_31_ffn | Add | 1 × 131072 × 8192 |
| 190 | RMSNorm_32_1 | RMSNorm | 1 × 131072 × 8192 |
| 191 | Attention_32 | Grouped Query Attn | 1 × 131072 × 8192 |
| 192 | Add_32_attn | Add | 1 × 131072 × 8192 |
| 193 | RMSNorm_32_2 | RMSNorm | 1 × 131072 × 8192 |
| 194 | FFN_32 | SwiGLU | 1 × 131072 × 8192 |
| 195 | Add_32_ffn | Add | 1 × 131072 × 8192 |
| 196 | RMSNorm_33_1 | RMSNorm | 1 × 131072 × 8192 |
| 197 | Attention_33 | Grouped Query Attn | 1 × 131072 × 8192 |
| 198 | Add_33_attn | Add | 1 × 131072 × 8192 |
| 199 | RMSNorm_33_2 | RMSNorm | 1 × 131072 × 8192 |
| 200 | FFN_33 | SwiGLU | 1 × 131072 × 8192 |
| 201 | Add_33_ffn | Add | 1 × 131072 × 8192 |
| 202 | RMSNorm_34_1 | RMSNorm | 1 × 131072 × 8192 |
| 203 | Attention_34 | Grouped Query Attn | 1 × 131072 × 8192 |
| 204 | Add_34_attn | Add | 1 × 131072 × 8192 |
| 205 | RMSNorm_34_2 | RMSNorm | 1 × 131072 × 8192 |
| 206 | FFN_34 | SwiGLU | 1 × 131072 × 8192 |
| 207 | Add_34_ffn | Add | 1 × 131072 × 8192 |
| 208 | RMSNorm_35_1 | RMSNorm | 1 × 131072 × 8192 |
| 209 | Attention_35 | Grouped Query Attn | 1 × 131072 × 8192 |
| 210 | Add_35_attn | Add | 1 × 131072 × 8192 |
| 211 | RMSNorm_35_2 | RMSNorm | 1 × 131072 × 8192 |
| 212 | FFN_35 | SwiGLU | 1 × 131072 × 8192 |
| 213 | Add_35_ffn | Add | 1 × 131072 × 8192 |
| 214 | RMSNorm_36_1 | RMSNorm | 1 × 131072 × 8192 |
| 215 | Attention_36 | Grouped Query Attn | 1 × 131072 × 8192 |
| 216 | Add_36_attn | Add | 1 × 131072 × 8192 |
| 217 | RMSNorm_36_2 | RMSNorm | 1 × 131072 × 8192 |
| 218 | FFN_36 | SwiGLU | 1 × 131072 × 8192 |
| 219 | Add_36_ffn | Add | 1 × 131072 × 8192 |
| 220 | RMSNorm_37_1 | RMSNorm | 1 × 131072 × 8192 |
| 221 | Attention_37 | Grouped Query Attn | 1 × 131072 × 8192 |
| 222 | Add_37_attn | Add | 1 × 131072 × 8192 |
| 223 | RMSNorm_37_2 | RMSNorm | 1 × 131072 × 8192 |
| 224 | FFN_37 | SwiGLU | 1 × 131072 × 8192 |
| 225 | Add_37_ffn | Add | 1 × 131072 × 8192 |
| 226 | RMSNorm_38_1 | RMSNorm | 1 × 131072 × 8192 |
| 227 | Attention_38 | Grouped Query Attn | 1 × 131072 × 8192 |
| 228 | Add_38_attn | Add | 1 × 131072 × 8192 |
| 229 | RMSNorm_38_2 | RMSNorm | 1 × 131072 × 8192 |
| 230 | FFN_38 | SwiGLU | 1 × 131072 × 8192 |
| 231 | Add_38_ffn | Add | 1 × 131072 × 8192 |
| 232 | RMSNorm_39_1 | RMSNorm | 1 × 131072 × 8192 |
| 233 | Attention_39 | Grouped Query Attn | 1 × 131072 × 8192 |
| 234 | Add_39_attn | Add | 1 × 131072 × 8192 |
| 235 | RMSNorm_39_2 | RMSNorm | 1 × 131072 × 8192 |
| 236 | FFN_39 | SwiGLU | 1 × 131072 × 8192 |
| 237 | Add_39_ffn | Add | 1 × 131072 × 8192 |
| 238 | RMSNorm_40_1 | RMSNorm | 1 × 131072 × 8192 |
| 239 | Attention_40 | Grouped Query Attn | 1 × 131072 × 8192 |
| 240 | Add_40_attn | Add | 1 × 131072 × 8192 |
| 241 | RMSNorm_40_2 | RMSNorm | 1 × 131072 × 8192 |
| 242 | FFN_40 | SwiGLU | 1 × 131072 × 8192 |
| 243 | Add_40_ffn | Add | 1 × 131072 × 8192 |
| 244 | RMSNorm_41_1 | RMSNorm | 1 × 131072 × 8192 |
| 245 | Attention_41 | Grouped Query Attn | 1 × 131072 × 8192 |
| 246 | Add_41_attn | Add | 1 × 131072 × 8192 |
| 247 | RMSNorm_41_2 | RMSNorm | 1 × 131072 × 8192 |
| 248 | FFN_41 | SwiGLU | 1 × 131072 × 8192 |
| 249 | Add_41_ffn | Add | 1 × 131072 × 8192 |
| 250 | RMSNorm_42_1 | RMSNorm | 1 × 131072 × 8192 |
| 251 | Attention_42 | Grouped Query Attn | 1 × 131072 × 8192 |
| 252 | Add_42_attn | Add | 1 × 131072 × 8192 |
| 253 | RMSNorm_42_2 | RMSNorm | 1 × 131072 × 8192 |
| 254 | FFN_42 | SwiGLU | 1 × 131072 × 8192 |
| 255 | Add_42_ffn | Add | 1 × 131072 × 8192 |
| 256 | RMSNorm_43_1 | RMSNorm | 1 × 131072 × 8192 |
| 257 | Attention_43 | Grouped Query Attn | 1 × 131072 × 8192 |
| 258 | Add_43_attn | Add | 1 × 131072 × 8192 |
| 259 | RMSNorm_43_2 | RMSNorm | 1 × 131072 × 8192 |
| 260 | FFN_43 | SwiGLU | 1 × 131072 × 8192 |
| 261 | Add_43_ffn | Add | 1 × 131072 × 8192 |
| 262 | RMSNorm_44_1 | RMSNorm | 1 × 131072 × 8192 |
| 263 | Attention_44 | Grouped Query Attn | 1 × 131072 × 8192 |
| 264 | Add_44_attn | Add | 1 × 131072 × 8192 |
| 265 | RMSNorm_44_2 | RMSNorm | 1 × 131072 × 8192 |
| 266 | FFN_44 | SwiGLU | 1 × 131072 × 8192 |
| 267 | Add_44_ffn | Add | 1 × 131072 × 8192 |
| 268 | RMSNorm_45_1 | RMSNorm | 1 × 131072 × 8192 |
| 269 | Attention_45 | Grouped Query Attn | 1 × 131072 × 8192 |
| 270 | Add_45_attn | Add | 1 × 131072 × 8192 |
| 271 | RMSNorm_45_2 | RMSNorm | 1 × 131072 × 8192 |
| 272 | FFN_45 | SwiGLU | 1 × 131072 × 8192 |
| 273 | Add_45_ffn | Add | 1 × 131072 × 8192 |
| 274 | RMSNorm_46_1 | RMSNorm | 1 × 131072 × 8192 |
| 275 | Attention_46 | Grouped Query Attn | 1 × 131072 × 8192 |
| 276 | Add_46_attn | Add | 1 × 131072 × 8192 |
| 277 | RMSNorm_46_2 | RMSNorm | 1 × 131072 × 8192 |
| 278 | FFN_46 | SwiGLU | 1 × 131072 × 8192 |
| 279 | Add_46_ffn | Add | 1 × 131072 × 8192 |
| 280 | RMSNorm_47_1 | RMSNorm | 1 × 131072 × 8192 |
| 281 | Attention_47 | Grouped Query Attn | 1 × 131072 × 8192 |
| 282 | Add_47_attn | Add | 1 × 131072 × 8192 |
| 283 | RMSNorm_47_2 | RMSNorm | 1 × 131072 × 8192 |
| 284 | FFN_47 | SwiGLU | 1 × 131072 × 8192 |
| 285 | Add_47_ffn | Add | 1 × 131072 × 8192 |
| 286 | RMSNorm_48_1 | RMSNorm | 1 × 131072 × 8192 |
| 287 | Attention_48 | Grouped Query Attn | 1 × 131072 × 8192 |
| 288 | Add_48_attn | Add | 1 × 131072 × 8192 |
| 289 | RMSNorm_48_2 | RMSNorm | 1 × 131072 × 8192 |
| 290 | FFN_48 | SwiGLU | 1 × 131072 × 8192 |
| 291 | Add_48_ffn | Add | 1 × 131072 × 8192 |
| 292 | RMSNorm_49_1 | RMSNorm | 1 × 131072 × 8192 |
| 293 | Attention_49 | Grouped Query Attn | 1 × 131072 × 8192 |
| 294 | Add_49_attn | Add | 1 × 131072 × 8192 |
| 295 | RMSNorm_49_2 | RMSNorm | 1 × 131072 × 8192 |
| 296 | FFN_49 | SwiGLU | 1 × 131072 × 8192 |
| 297 | Add_49_ffn | Add | 1 × 131072 × 8192 |
| 298 | RMSNorm_50_1 | RMSNorm | 1 × 131072 × 8192 |
| 299 | Attention_50 | Grouped Query Attn | 1 × 131072 × 8192 |
| 300 | Add_50_attn | Add | 1 × 131072 × 8192 |
| 301 | RMSNorm_50_2 | RMSNorm | 1 × 131072 × 8192 |
| 302 | FFN_50 | SwiGLU | 1 × 131072 × 8192 |
| 303 | Add_50_ffn | Add | 1 × 131072 × 8192 |
| 304 | RMSNorm_51_1 | RMSNorm | 1 × 131072 × 8192 |
| 305 | Attention_51 | Grouped Query Attn | 1 × 131072 × 8192 |
| 306 | Add_51_attn | Add | 1 × 131072 × 8192 |
| 307 | RMSNorm_51_2 | RMSNorm | 1 × 131072 × 8192 |
| 308 | FFN_51 | SwiGLU | 1 × 131072 × 8192 |
| 309 | Add_51_ffn | Add | 1 × 131072 × 8192 |
| 310 | RMSNorm_52_1 | RMSNorm | 1 × 131072 × 8192 |
| 311 | Attention_52 | Grouped Query Attn | 1 × 131072 × 8192 |
| 312 | Add_52_attn | Add | 1 × 131072 × 8192 |
| 313 | RMSNorm_52_2 | RMSNorm | 1 × 131072 × 8192 |
| 314 | FFN_52 | SwiGLU | 1 × 131072 × 8192 |
| 315 | Add_52_ffn | Add | 1 × 131072 × 8192 |
| 316 | RMSNorm_53_1 | RMSNorm | 1 × 131072 × 8192 |
| 317 | Attention_53 | Grouped Query Attn | 1 × 131072 × 8192 |
| 318 | Add_53_attn | Add | 1 × 131072 × 8192 |
| 319 | RMSNorm_53_2 | RMSNorm | 1 × 131072 × 8192 |
| 320 | FFN_53 | SwiGLU | 1 × 131072 × 8192 |
| 321 | Add_53_ffn | Add | 1 × 131072 × 8192 |
| 322 | RMSNorm_54_1 | RMSNorm | 1 × 131072 × 8192 |
| 323 | Attention_54 | Grouped Query Attn | 1 × 131072 × 8192 |
| 324 | Add_54_attn | Add | 1 × 131072 × 8192 |
| 325 | RMSNorm_54_2 | RMSNorm | 1 × 131072 × 8192 |
| 326 | FFN_54 | SwiGLU | 1 × 131072 × 8192 |
| 327 | Add_54_ffn | Add | 1 × 131072 × 8192 |
| 328 | RMSNorm_55_1 | RMSNorm | 1 × 131072 × 8192 |
| 329 | Attention_55 | Grouped Query Attn | 1 × 131072 × 8192 |
| 330 | Add_55_attn | Add | 1 × 131072 × 8192 |
| 331 | RMSNorm_55_2 | RMSNorm | 1 × 131072 × 8192 |
| 332 | FFN_55 | SwiGLU | 1 × 131072 × 8192 |
| 333 | Add_55_ffn | Add | 1 × 131072 × 8192 |
| 334 | RMSNorm_56_1 | RMSNorm | 1 × 131072 × 8192 |
| 335 | Attention_56 | Grouped Query Attn | 1 × 131072 × 8192 |
| 336 | Add_56_attn | Add | 1 × 131072 × 8192 |
| 337 | RMSNorm_56_2 | RMSNorm | 1 × 131072 × 8192 |
| 338 | FFN_56 | SwiGLU | 1 × 131072 × 8192 |
| 339 | Add_56_ffn | Add | 1 × 131072 × 8192 |
| 340 | RMSNorm_57_1 | RMSNorm | 1 × 131072 × 8192 |
| 341 | Attention_57 | Grouped Query Attn | 1 × 131072 × 8192 |
| 342 | Add_57_attn | Add | 1 × 131072 × 8192 |
| 343 | RMSNorm_57_2 | RMSNorm | 1 × 131072 × 8192 |
| 344 | FFN_57 | SwiGLU | 1 × 131072 × 8192 |
| 345 | Add_57_ffn | Add | 1 × 131072 × 8192 |
| 346 | RMSNorm_58_1 | RMSNorm | 1 × 131072 × 8192 |
| 347 | Attention_58 | Grouped Query Attn | 1 × 131072 × 8192 |
| 348 | Add_58_attn | Add | 1 × 131072 × 8192 |
| 349 | RMSNorm_58_2 | RMSNorm | 1 × 131072 × 8192 |
| 350 | FFN_58 | SwiGLU | 1 × 131072 × 8192 |
| 351 | Add_58_ffn | Add | 1 × 131072 × 8192 |
| 352 | RMSNorm_59_1 | RMSNorm | 1 × 131072 × 8192 |
| 353 | Attention_59 | Grouped Query Attn | 1 × 131072 × 8192 |
| 354 | Add_59_attn | Add | 1 × 131072 × 8192 |
| 355 | RMSNorm_59_2 | RMSNorm | 1 × 131072 × 8192 |
| 356 | FFN_59 | SwiGLU | 1 × 131072 × 8192 |
| 357 | Add_59_ffn | Add | 1 × 131072 × 8192 |
| 358 | RMSNorm_60_1 | RMSNorm | 1 × 131072 × 8192 |
| 359 | Attention_60 | Grouped Query Attn | 1 × 131072 × 8192 |
| 360 | Add_60_attn | Add | 1 × 131072 × 8192 |
| 361 | RMSNorm_60_2 | RMSNorm | 1 × 131072 × 8192 |
| 362 | FFN_60 | SwiGLU | 1 × 131072 × 8192 |
| 363 | Add_60_ffn | Add | 1 × 131072 × 8192 |
| 364 | RMSNorm_61_1 | RMSNorm | 1 × 131072 × 8192 |
| 365 | Attention_61 | Grouped Query Attn | 1 × 131072 × 8192 |
| 366 | Add_61_attn | Add | 1 × 131072 × 8192 |
| 367 | RMSNorm_61_2 | RMSNorm | 1 × 131072 × 8192 |
| 368 | FFN_61 | SwiGLU | 1 × 131072 × 8192 |
| 369 | Add_61_ffn | Add | 1 × 131072 × 8192 |
| 370 | RMSNorm_62_1 | RMSNorm | 1 × 131072 × 8192 |
| 371 | Attention_62 | Grouped Query Attn | 1 × 131072 × 8192 |
| 372 | Add_62_attn | Add | 1 × 131072 × 8192 |
| 373 | RMSNorm_62_2 | RMSNorm | 1 × 131072 × 8192 |
| 374 | FFN_62 | SwiGLU | 1 × 131072 × 8192 |
| 375 | Add_62_ffn | Add | 1 × 131072 × 8192 |
| 376 | RMSNorm_63_1 | RMSNorm | 1 × 131072 × 8192 |
| 377 | Attention_63 | Grouped Query Attn | 1 × 131072 × 8192 |
| 378 | Add_63_attn | Add | 1 × 131072 × 8192 |
| 379 | RMSNorm_63_2 | RMSNorm | 1 × 131072 × 8192 |
| 380 | FFN_63 | SwiGLU | 1 × 131072 × 8192 |
| 381 | Add_63_ffn | Add | 1 × 131072 × 8192 |
| 382 | RMSNorm_64_1 | RMSNorm | 1 × 131072 × 8192 |
| 383 | Attention_64 | Grouped Query Attn | 1 × 131072 × 8192 |
| 384 | Add_64_attn | Add | 1 × 131072 × 8192 |
| 385 | RMSNorm_64_2 | RMSNorm | 1 × 131072 × 8192 |
| 386 | FFN_64 | SwiGLU | 1 × 131072 × 8192 |
| 387 | Add_64_ffn | Add | 1 × 131072 × 8192 |
| 388 | RMSNorm_65_1 | RMSNorm | 1 × 131072 × 8192 |
| 389 | Attention_65 | Grouped Query Attn | 1 × 131072 × 8192 |
| 390 | Add_65_attn | Add | 1 × 131072 × 8192 |
| 391 | RMSNorm_65_2 | RMSNorm | 1 × 131072 × 8192 |
| 392 | FFN_65 | SwiGLU | 1 × 131072 × 8192 |
| 393 | Add_65_ffn | Add | 1 × 131072 × 8192 |
| 394 | RMSNorm_66_1 | RMSNorm | 1 × 131072 × 8192 |
| 395 | Attention_66 | Grouped Query Attn | 1 × 131072 × 8192 |
| 396 | Add_66_attn | Add | 1 × 131072 × 8192 |
| 397 | RMSNorm_66_2 | RMSNorm | 1 × 131072 × 8192 |
| 398 | FFN_66 | SwiGLU | 1 × 131072 × 8192 |
| 399 | Add_66_ffn | Add | 1 × 131072 × 8192 |
| 400 | RMSNorm_67_1 | RMSNorm | 1 × 131072 × 8192 |
| 401 | Attention_67 | Grouped Query Attn | 1 × 131072 × 8192 |
| 402 | Add_67_attn | Add | 1 × 131072 × 8192 |
| 403 | RMSNorm_67_2 | RMSNorm | 1 × 131072 × 8192 |
| 404 | FFN_67 | SwiGLU | 1 × 131072 × 8192 |
| 405 | Add_67_ffn | Add | 1 × 131072 × 8192 |
| 406 | RMSNorm_68_1 | RMSNorm | 1 × 131072 × 8192 |
| 407 | Attention_68 | Grouped Query Attn | 1 × 131072 × 8192 |
| 408 | Add_68_attn | Add | 1 × 131072 × 8192 |
| 409 | RMSNorm_68_2 | RMSNorm | 1 × 131072 × 8192 |
| 410 | FFN_68 | SwiGLU | 1 × 131072 × 8192 |
| 411 | Add_68_ffn | Add | 1 × 131072 × 8192 |
| 412 | RMSNorm_69_1 | RMSNorm | 1 × 131072 × 8192 |
| 413 | Attention_69 | Grouped Query Attn | 1 × 131072 × 8192 |
| 414 | Add_69_attn | Add | 1 × 131072 × 8192 |
| 415 | RMSNorm_69_2 | RMSNorm | 1 × 131072 × 8192 |
| 416 | FFN_69 | SwiGLU | 1 × 131072 × 8192 |
| 417 | Add_69_ffn | Add | 1 × 131072 × 8192 |
| 418 | RMSNorm_70_1 | RMSNorm | 1 × 131072 × 8192 |
| 419 | Attention_70 | Grouped Query Attn | 1 × 131072 × 8192 |
| 420 | Add_70_attn | Add | 1 × 131072 × 8192 |
| 421 | RMSNorm_70_2 | RMSNorm | 1 × 131072 × 8192 |
| 422 | FFN_70 | SwiGLU | 1 × 131072 × 8192 |
| 423 | Add_70_ffn | Add | 1 × 131072 × 8192 |
| 424 | RMSNorm_71_1 | RMSNorm | 1 × 131072 × 8192 |
| 425 | Attention_71 | Grouped Query Attn | 1 × 131072 × 8192 |
| 426 | Add_71_attn | Add | 1 × 131072 × 8192 |
| 427 | RMSNorm_71_2 | RMSNorm | 1 × 131072 × 8192 |
| 428 | FFN_71 | SwiGLU | 1 × 131072 × 8192 |
| 429 | Add_71_ffn | Add | 1 × 131072 × 8192 |
| 430 | RMSNorm_72_1 | RMSNorm | 1 × 131072 × 8192 |
| 431 | Attention_72 | Grouped Query Attn | 1 × 131072 × 8192 |
| 432 | Add_72_attn | Add | 1 × 131072 × 8192 |
| 433 | RMSNorm_72_2 | RMSNorm | 1 × 131072 × 8192 |
| 434 | FFN_72 | SwiGLU | 1 × 131072 × 8192 |
| 435 | Add_72_ffn | Add | 1 × 131072 × 8192 |
| 436 | RMSNorm_73_1 | RMSNorm | 1 × 131072 × 8192 |
| 437 | Attention_73 | Grouped Query Attn | 1 × 131072 × 8192 |
| 438 | Add_73_attn | Add | 1 × 131072 × 8192 |
| 439 | RMSNorm_73_2 | RMSNorm | 1 × 131072 × 8192 |
| 440 | FFN_73 | SwiGLU | 1 × 131072 × 8192 |
| 441 | Add_73_ffn | Add | 1 × 131072 × 8192 |
| 442 | RMSNorm_74_1 | RMSNorm | 1 × 131072 × 8192 |
| 443 | Attention_74 | Grouped Query Attn | 1 × 131072 × 8192 |
| 444 | Add_74_attn | Add | 1 × 131072 × 8192 |
| 445 | RMSNorm_74_2 | RMSNorm | 1 × 131072 × 8192 |
| 446 | FFN_74 | SwiGLU | 1 × 131072 × 8192 |
| 447 | Add_74_ffn | Add | 1 × 131072 × 8192 |
| 448 | RMSNorm_75_1 | RMSNorm | 1 × 131072 × 8192 |
| 449 | Attention_75 | Grouped Query Attn | 1 × 131072 × 8192 |
| 450 | Add_75_attn | Add | 1 × 131072 × 8192 |
| 451 | RMSNorm_75_2 | RMSNorm | 1 × 131072 × 8192 |
| 452 | FFN_75 | SwiGLU | 1 × 131072 × 8192 |
| 453 | Add_75_ffn | Add | 1 × 131072 × 8192 |
| 454 | RMSNorm_76_1 | RMSNorm | 1 × 131072 × 8192 |
| 455 | Attention_76 | Grouped Query Attn | 1 × 131072 × 8192 |
| 456 | Add_76_attn | Add | 1 × 131072 × 8192 |
| 457 | RMSNorm_76_2 | RMSNorm | 1 × 131072 × 8192 |
| 458 | FFN_76 | SwiGLU | 1 × 131072 × 8192 |
| 459 | Add_76_ffn | Add | 1 × 131072 × 8192 |
| 460 | RMSNorm_77_1 | RMSNorm | 1 × 131072 × 8192 |
| 461 | Attention_77 | Grouped Query Attn | 1 × 131072 × 8192 |
| 462 | Add_77_attn | Add | 1 × 131072 × 8192 |
| 463 | RMSNorm_77_2 | RMSNorm | 1 × 131072 × 8192 |
| 464 | FFN_77 | SwiGLU | 1 × 131072 × 8192 |
| 465 | Add_77_ffn | Add | 1 × 131072 × 8192 |
| 466 | RMSNorm_78_1 | RMSNorm | 1 × 131072 × 8192 |
| 467 | Attention_78 | Grouped Query Attn | 1 × 131072 × 8192 |
| 468 | Add_78_attn | Add | 1 × 131072 × 8192 |
| 469 | RMSNorm_78_2 | RMSNorm | 1 × 131072 × 8192 |
| 470 | FFN_78 | SwiGLU | 1 × 131072 × 8192 |
| 471 | Add_78_ffn | Add | 1 × 131072 × 8192 |
| 472 | RMSNorm_79_1 | RMSNorm | 1 × 131072 × 8192 |
| 473 | Attention_79 | Grouped Query Attn | 1 × 131072 × 8192 |
| 474 | Add_79_attn | Add | 1 × 131072 × 8192 |
| 475 | RMSNorm_79_2 | RMSNorm | 1 × 131072 × 8192 |
| 476 | FFN_79 | SwiGLU | 1 × 131072 × 8192 |
| 477 | Add_79_ffn | Add | 1 × 131072 × 8192 |
| 478 | RMSNorm_80_1 | RMSNorm | 1 × 131072 × 8192 |
| 479 | Attention_80 | Grouped Query Attn | 1 × 131072 × 8192 |
| 480 | Add_80_attn | Add | 1 × 131072 × 8192 |
| 481 | RMSNorm_80_2 | RMSNorm | 1 × 131072 × 8192 |
| 482 | FFN_80 | SwiGLU | 1 × 131072 × 8192 |
| 483 | Add_80_ffn | Add | 1 × 131072 × 8192 |
| 484 | Final_RMSNorm | RMSNorm | 1 × 131072 × 8192 |
| 485 | LM_Head | Linear | 1 × 131072 × 128256 |
| 486 | Output | Output | 1 × 131072 × 128256 |

## Do this to your own model

```
pip install neurarch-trace
neurarch-trace nvidia/Llama-3_3-Nemotron-Super-49B-v1_5-FP8 --plan --share
```

- Page: https://neurarch.com/m/llama-3-3-nemotron-super-49b-v1-5-fp8.html
- All models: https://neurarch.com/m/index.json
- The checks: https://neurarch.com/r/index.json