Models / nemotron_h

NVIDIA-Nemotron-3-Nano-30B-A3B-BF16: GPU memory to fine-tune

Fine-tuning NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 at batch 1 and 2048 tokens takes about 614 GB of GPU memory for a full fine-tune (at least 6 x H200 sharded), and 70.9 GB with LoRA r=16 on a bf16 base (fits one H200 141 GB). Estimated from the model's structure, reading the config under transformers semantics, not measured. What fits on which card.

Reconstructed from its own config.json with no weights read. 845K downloads on Hugging Face.

Our count against the checkpoint

The left number comes from the graph. The right one is the number of scalars in the published weight files. Nothing on this page was tuned to make them agree.

Derived from structure

31.58B

31,578,068,416 parameters

In the published checkpoint

31.58B

31,577,937,344 scalars · safetensors.total, read 2026-08-24

Delta

+0.00%

custom-code This repository ships its own modeling code (`auto_map`, e.g. `configuration_nemotron_h.py`), so `config.json` names a class in the repo rather than an architecture `transformers` defines. The graph below is what those config keys mean under `transformers` semantics, which is not necessarily what the repo's own file builds. A gap here is a statement about what we read, not about the checkpoint.

What it costs to run

Cost is a roofline estimate on the priced GPU for 10 epochs at batch 32, 2048 tokens over 50,000 samples (assumed; no dataset attached).

Layers

159

Will it forward-pass

Yes

Priced on

A10G (24GB)

Est. one run

$295.32

Memory: what fits on which card

Largest batch per card. Training is sized at 2048 tokens per sample, serving at 4096 tokens per sequence.

Full fine-tune, batch 1 at 2048 tokens

614 GB

LoRA r=16, batch 1 at 2048 tokens

70.9 GB

Serve: bf16 weights

63.2 GB

Serve: KV cache, 4096 tokens

25 MB

Serve: recurrent state

25 MB

Card	Full fine-tune	LoRA r=16	Serve bf16, 4096 tokens
T4 16 GB	does not fit at batch 1	does not fit at batch 1	does not fit at batch 1
L4 24 GB	does not fit at batch 1	does not fit at batch 1	does not fit at batch 1
A100 40 GB	does not fit at batch 1	does not fit at batch 1	does not fit at batch 1
A100 / H100 80 GB	does not fit at batch 1	does not fit at batch 1	batch 127
H200 141 GB	does not fit at batch 1	batch 7	batch 1185
Sharded, by memory alone	at least 6 x H200 or 9 x 80 GB

Training: 2048 tokens per sample (2048, or the model's context when shorter; the structure table is at the config's maximum, 262144), AdamW with bf16 autocast, SDPA attention (no score matrix stored), 1.15x headroom on every card. Full fine-tune holds weights, gradients and optimizer state (1.2 x 16 bytes per parameter) plus activations. LoRA is r=16 on every linear layer but the head, base frozen in bf16 (2 bytes per parameter), adapters at full AdamW state, same activations. A two-card split is a contiguous layer cut (pipeline parallel), computed only where one card cannot hold batch 1. "Sharded" is a floor: the batch-1 peak with headroom divided across cards as if it split perfectly evenly (FSDP / ZeRO-3 for training, tensor parallel for serving), with no communication buffers counted. Serving: bf16 weights plus the KV cache of 4096 tokens per sequence plus the fixed recurrent state of the SSM / linear-attention layers; prefill working set, CUDA context and framework reserve not counted.

These are estimates from the graph, not measurements. Against measured runs, held out by architecture, the estimate flagged 113 of the 131 jobs that did not fit their card and said "fits" on the other 18. Every measured run.

Structure

161 nodes. Output shapes are propagated from the input shape, batch dimension excluded, at 262144 tokens: the config's maximum context, not the 2048 the memory above is sized at.

	Layer	Type	Output shape
1	Input	Input	1 × 262144
2	Embedding	Embedding	1 × 262144 × 2688
3	RMSNorm_1_1	RMSNorm	1 × 262144 × 2688
4	Mamba_1	Mamba-2 (SSD)	1 × 262144 × 2688
5	Add_1_attn	Add	1 × 262144 × 2688
6	RMSNorm_2_2	RMSNorm	1 × 262144 × 2688
7	MoE_2	Shared-Expert MoE	1 × 262144 × 2688
8	Add_2_ffn	Add	1 × 262144 × 2688
9	RMSNorm_3_1	RMSNorm	1 × 262144 × 2688
10	Mamba_3	Mamba-2 (SSD)	1 × 262144 × 2688
11	Add_3_attn	Add	1 × 262144 × 2688
12	RMSNorm_4_2	RMSNorm	1 × 262144 × 2688
13	MoE_4	Shared-Expert MoE	1 × 262144 × 2688
14	Add_4_ffn	Add	1 × 262144 × 2688
15	RMSNorm_5_1	RMSNorm	1 × 262144 × 2688
16	Mamba_5	Mamba-2 (SSD)	1 × 262144 × 2688
17	Add_5_attn	Add	1 × 262144 × 2688
18	RMSNorm_6_1	RMSNorm	1 × 262144 × 2688
19	Attention_6	Grouped Query Attn	1 × 262144 × 2688
20	Add_6_attn	Add	1 × 262144 × 2688
21	RMSNorm_7_2	RMSNorm	1 × 262144 × 2688
22	MoE_7	Shared-Expert MoE	1 × 262144 × 2688
23	Add_7_ffn	Add	1 × 262144 × 2688
24	RMSNorm_8_1	RMSNorm	1 × 262144 × 2688
25	Mamba_8	Mamba-2 (SSD)	1 × 262144 × 2688
26	Add_8_attn	Add	1 × 262144 × 2688
27	RMSNorm_9_2	RMSNorm	1 × 262144 × 2688
28	MoE_9	Shared-Expert MoE	1 × 262144 × 2688
29	Add_9_ffn	Add	1 × 262144 × 2688
30	RMSNorm_10_1	RMSNorm	1 × 262144 × 2688
31	Mamba_10	Mamba-2 (SSD)	1 × 262144 × 2688
32	Add_10_attn	Add	1 × 262144 × 2688
33	RMSNorm_11_2	RMSNorm	1 × 262144 × 2688
34	MoE_11	Shared-Expert MoE	1 × 262144 × 2688
35	Add_11_ffn	Add	1 × 262144 × 2688
36	RMSNorm_12_1	RMSNorm	1 × 262144 × 2688
37	Mamba_12	Mamba-2 (SSD)	1 × 262144 × 2688
38	Add_12_attn	Add	1 × 262144 × 2688
39	RMSNorm_13_1	RMSNorm	1 × 262144 × 2688
40	Attention_13	Grouped Query Attn	1 × 262144 × 2688
41	Add_13_attn	Add	1 × 262144 × 2688
42	RMSNorm_14_2	RMSNorm	1 × 262144 × 2688
43	MoE_14	Shared-Expert MoE	1 × 262144 × 2688
44	Add_14_ffn	Add	1 × 262144 × 2688
45	RMSNorm_15_1	RMSNorm	1 × 262144 × 2688
46	Mamba_15	Mamba-2 (SSD)	1 × 262144 × 2688
47	Add_15_attn	Add	1 × 262144 × 2688
48	RMSNorm_16_2	RMSNorm	1 × 262144 × 2688
49	MoE_16	Shared-Expert MoE	1 × 262144 × 2688
50	Add_16_ffn	Add	1 × 262144 × 2688
51	RMSNorm_17_1	RMSNorm	1 × 262144 × 2688
52	Mamba_17	Mamba-2 (SSD)	1 × 262144 × 2688
53	Add_17_attn	Add	1 × 262144 × 2688
54	RMSNorm_18_2	RMSNorm	1 × 262144 × 2688
55	MoE_18	Shared-Expert MoE	1 × 262144 × 2688
56	Add_18_ffn	Add	1 × 262144 × 2688
57	RMSNorm_19_1	RMSNorm	1 × 262144 × 2688
58	Mamba_19	Mamba-2 (SSD)	1 × 262144 × 2688
59	Add_19_attn	Add	1 × 262144 × 2688
60	RMSNorm_20_1	RMSNorm	1 × 262144 × 2688
61	Attention_20	Grouped Query Attn	1 × 262144 × 2688
62	Add_20_attn	Add	1 × 262144 × 2688
63	RMSNorm_21_2	RMSNorm	1 × 262144 × 2688
64	MoE_21	Shared-Expert MoE	1 × 262144 × 2688
65	Add_21_ffn	Add	1 × 262144 × 2688
66	RMSNorm_22_1	RMSNorm	1 × 262144 × 2688
67	Mamba_22	Mamba-2 (SSD)	1 × 262144 × 2688
68	Add_22_attn	Add	1 × 262144 × 2688
69	RMSNorm_23_2	RMSNorm	1 × 262144 × 2688
70	MoE_23	Shared-Expert MoE	1 × 262144 × 2688
71	Add_23_ffn	Add	1 × 262144 × 2688
72	RMSNorm_24_1	RMSNorm	1 × 262144 × 2688
73	Mamba_24	Mamba-2 (SSD)	1 × 262144 × 2688
74	Add_24_attn	Add	1 × 262144 × 2688
75	RMSNorm_25_2	RMSNorm	1 × 262144 × 2688
76	MoE_25	Shared-Expert MoE	1 × 262144 × 2688
77	Add_25_ffn	Add	1 × 262144 × 2688
78	RMSNorm_26_1	RMSNorm	1 × 262144 × 2688
79	Mamba_26	Mamba-2 (SSD)	1 × 262144 × 2688
80	Add_26_attn	Add	1 × 262144 × 2688
81	RMSNorm_27_1	RMSNorm	1 × 262144 × 2688
82	Attention_27	Grouped Query Attn	1 × 262144 × 2688
83	Add_27_attn	Add	1 × 262144 × 2688
84	RMSNorm_28_2	RMSNorm	1 × 262144 × 2688
85	MoE_28	Shared-Expert MoE	1 × 262144 × 2688
86	Add_28_ffn	Add	1 × 262144 × 2688
87	RMSNorm_29_1	RMSNorm	1 × 262144 × 2688
88	Mamba_29	Mamba-2 (SSD)	1 × 262144 × 2688
89	Add_29_attn	Add	1 × 262144 × 2688
90	RMSNorm_30_2	RMSNorm	1 × 262144 × 2688
91	MoE_30	Shared-Expert MoE	1 × 262144 × 2688
92	Add_30_ffn	Add	1 × 262144 × 2688
93	RMSNorm_31_1	RMSNorm	1 × 262144 × 2688
94	Mamba_31	Mamba-2 (SSD)	1 × 262144 × 2688
95	Add_31_attn	Add	1 × 262144 × 2688
96	RMSNorm_32_2	RMSNorm	1 × 262144 × 2688
97	MoE_32	Shared-Expert MoE	1 × 262144 × 2688
98	Add_32_ffn	Add	1 × 262144 × 2688
99	RMSNorm_33_1	RMSNorm	1 × 262144 × 2688
100	Mamba_33	Mamba-2 (SSD)	1 × 262144 × 2688
101	Add_33_attn	Add	1 × 262144 × 2688
102	RMSNorm_34_1	RMSNorm	1 × 262144 × 2688
103	Attention_34	Grouped Query Attn	1 × 262144 × 2688
104	Add_34_attn	Add	1 × 262144 × 2688
105	RMSNorm_35_2	RMSNorm	1 × 262144 × 2688
106	MoE_35	Shared-Expert MoE	1 × 262144 × 2688
107	Add_35_ffn	Add	1 × 262144 × 2688
108	RMSNorm_36_1	RMSNorm	1 × 262144 × 2688
109	Mamba_36	Mamba-2 (SSD)	1 × 262144 × 2688
110	Add_36_attn	Add	1 × 262144 × 2688
111	RMSNorm_37_2	RMSNorm	1 × 262144 × 2688
112	MoE_37	Shared-Expert MoE	1 × 262144 × 2688
113	Add_37_ffn	Add	1 × 262144 × 2688
114	RMSNorm_38_1	RMSNorm	1 × 262144 × 2688
115	Mamba_38	Mamba-2 (SSD)	1 × 262144 × 2688
116	Add_38_attn	Add	1 × 262144 × 2688
117	RMSNorm_39_2	RMSNorm	1 × 262144 × 2688
118	MoE_39	Shared-Expert MoE	1 × 262144 × 2688
119	Add_39_ffn	Add	1 × 262144 × 2688
120	RMSNorm_40_1	RMSNorm	1 × 262144 × 2688
121	Mamba_40	Mamba-2 (SSD)	1 × 262144 × 2688
122	Add_40_attn	Add	1 × 262144 × 2688
123	RMSNorm_41_2	RMSNorm	1 × 262144 × 2688
124	MoE_41	Shared-Expert MoE	1 × 262144 × 2688
125	Add_41_ffn	Add	1 × 262144 × 2688
126	RMSNorm_42_1	RMSNorm	1 × 262144 × 2688
127	Mamba_42	Mamba-2 (SSD)	1 × 262144 × 2688
128	Add_42_attn	Add	1 × 262144 × 2688
129	RMSNorm_43_1	RMSNorm	1 × 262144 × 2688
130	Attention_43	Grouped Query Attn	1 × 262144 × 2688
131	Add_43_attn	Add	1 × 262144 × 2688
132	RMSNorm_44_2	RMSNorm	1 × 262144 × 2688
133	MoE_44	Shared-Expert MoE	1 × 262144 × 2688
134	Add_44_ffn	Add	1 × 262144 × 2688
135	RMSNorm_45_1	RMSNorm	1 × 262144 × 2688
136	Mamba_45	Mamba-2 (SSD)	1 × 262144 × 2688
137	Add_45_attn	Add	1 × 262144 × 2688
138	RMSNorm_46_2	RMSNorm	1 × 262144 × 2688
139	MoE_46	Shared-Expert MoE	1 × 262144 × 2688
140	Add_46_ffn	Add	1 × 262144 × 2688
141	RMSNorm_47_1	RMSNorm	1 × 262144 × 2688
142	Mamba_47	Mamba-2 (SSD)	1 × 262144 × 2688
143	Add_47_attn	Add	1 × 262144 × 2688
144	RMSNorm_48_2	RMSNorm	1 × 262144 × 2688
145	MoE_48	Shared-Expert MoE	1 × 262144 × 2688
146	Add_48_ffn	Add	1 × 262144 × 2688
147	RMSNorm_49_1	RMSNorm	1 × 262144 × 2688
148	Mamba_49	Mamba-2 (SSD)	1 × 262144 × 2688
149	Add_49_attn	Add	1 × 262144 × 2688
150	RMSNorm_50_2	RMSNorm	1 × 262144 × 2688
151	MoE_50	Shared-Expert MoE	1 × 262144 × 2688
152	Add_50_ffn	Add	1 × 262144 × 2688
153	RMSNorm_51_1	RMSNorm	1 × 262144 × 2688
154	Mamba_51	Mamba-2 (SSD)	1 × 262144 × 2688
155	Add_51_attn	Add	1 × 262144 × 2688
156	RMSNorm_52_2	RMSNorm	1 × 262144 × 2688
157	MoE_52	Shared-Expert MoE	1 × 262144 × 2688
158	Add_52_ffn	Add	1 × 262144 × 2688
159	Final_RMSNorm	RMSNorm	1 × 262144 × 2688
160	LM_Head	Linear	1 × 262144 × 131072
161	Output	Output	1 × 262144 × 131072

What the verifier says

warn6 attention layer(s) present but no positional encoding found. Attention is permutation-invariant, without position information the model cannot distinguish token order. Fix: Add a PositionalEncoding (sinusoidal) or RoPE layer before the first attention layer.
attention-no-pe

What changed upstream

The change tracker follows nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16: every revision it has read, what changed in the config, tokenizer, chat template or weights between them, and an OSV advisory per change.

Revision history and changes Badge JSON OSV advisories All tracked changes

Do this to your own model

Same numbers, on a model in your repo, in one command. No account.

pip install neurarch-trace
neurarch-trace nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 --plan --share

Paste a model instead Markdown twin Index JSON POST /api/v1/plan

Other nemotron_h checkpoints

NVIDIA-Nemotron-3-Nano-30B-A3B-FP8

31.58B derived · +0.00% against the checkpoint

NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4

31.58B derived · +73.1% against the checkpoint

NVIDIA-Nemotron-3-Super-120B-A12B-BF16

124B derived · +0.00% against the checkpoint

NVIDIA-Nemotron-3-Super-120B-A12B-FP8

124B derived · +0.00% against the checkpoint