Comparisons / Diffusion UNet vs DiT-XL/2

Diffusion UNet vs DiT-XL/2

Convolutional against transformer backbones for diffusion.

DiT-XL/2 has 664M more parameters than Diffusion UNet: 199 layers added, 14 removed, 4 changed.

Baseline

Diffusion UNet

Layers: 17
Parameters: 6.7M
Input: 4 × 64 × 64
Output: 4 × 64 × 64
Forward-passes: yes
Est. train cost: $0.541

Compared

DiT-XL/2

Layers: 201
Parameters: 671M
Input: 4 × 32 × 32
Output: 256 × 32
Forward-passes: yes
Est. train cost: $4.41

The deltas

Every number is DiT-XL/2 relative to Diffusion UNet.

Parameters

+664M (100× the size)

Layers

+184

Added

199

Removed

Changed

Unchanged

Which GPUs each one fits

Each side is measured at its own declared input (4 × 64 × 64 against 4 × 32 × 32). Both columns are right about their own model; the difference between them is not a fact about the designs.

GPU	Diffusion UNet	DiT-XL/2
T4 16GB	fits	no
A100 40GB	fits	fits
H100 80GB	fits	fits

Layer by layer

Aligned in topological order. 1 of 218 rows are the same layer with the same parameters.

Show all 218 rows

		Diffusion UNet			DiT-XL/2
		Layer	Params	Output	Layer	Params	Output
1	changed shape	noisy_latent Input		4 × 64 × 64	noisy_latent Input		4 × 32 × 32
2	removed	conv_in Conv2d	3.2K	320 × 64 × 64	—
3	added	—			timestep_+_class Input		1 × 2
4	added	—			patchify_2x2 Patch Embed	20K	256 × 1152
5	added	—			cond_embed Embedding	1.2M	1 × 2 × 1152
6	added	—			pos_embed Positional Encoding		256 × 1152
7	added	—			adaLN_1 Linear	8.0M	1 × 2 × 6912
8	added	—			adaLN_2 Linear	8.0M	1 × 2 × 6912
9	added	—			adaLN_3 Linear	8.0M	1 × 2 × 6912
10	added	—			adaLN_4 Linear	8.0M	1 × 2 × 6912
11	added	—			adaLN_5 Linear	8.0M	1 × 2 × 6912
12	added	—			adaLN_6 Linear	8.0M	1 × 2 × 6912
13	added	—			adaLN_7 Linear	8.0M	1 × 2 × 6912
14	added	—			adaLN_8 Linear	8.0M	1 × 2 × 6912
15	added	—			adaLN_9 Linear	8.0M	1 × 2 × 6912
16	added	—			adaLN_10 Linear	8.0M	1 × 2 × 6912
17	added	—			adaLN_11 Linear	8.0M	1 × 2 × 6912
18	added	—			adaLN_12 Linear	8.0M	1 × 2 × 6912
19	added	—			adaLN_13 Linear	8.0M	1 × 2 × 6912
20	added	—			adaLN_14 Linear	8.0M	1 × 2 × 6912
21	added	—			adaLN_15 Linear	8.0M	1 × 2 × 6912
22	added	—			adaLN_16 Linear	8.0M	1 × 2 × 6912
23	added	—			adaLN_17 Linear	8.0M	1 × 2 × 6912
24	added	—			adaLN_18 Linear	8.0M	1 × 2 × 6912
25	added	—			adaLN_19 Linear	8.0M	1 × 2 × 6912
26	added	—			adaLN_20 Linear	8.0M	1 × 2 × 6912
27	added	—			adaLN_21 Linear	8.0M	1 × 2 × 6912
28	added	—			adaLN_22 Linear	8.0M	1 × 2 × 6912
29	added	—			adaLN_23 Linear	8.0M	1 × 2 × 6912
30	added	—			adaLN_24 Linear	8.0M	1 × 2 × 6912
31	added	—			adaLN_25 Linear	8.0M	1 × 2 × 6912
32	added	—			adaLN_26 Linear	8.0M	1 × 2 × 6912
33	added	—			adaLN_27 Linear	8.0M	1 × 2 × 6912
34	added	—			adaLN_28 Linear	8.0M	1 × 2 × 6912
35	changed type, numGroups, numChannels, normalizedShape	down1_norm Group Norm	640	320 × 64 × 64	norm1_1 Layer Norm	2.3K	256 × 1152
36	removed	down1_conv Conv2d	3.2K	320 × 64 × 64	—
37	removed	down1_silu Swish		320 × 64 × 64	—
38	removed	to_tokens Reshape		4096 × 320	—
39	removed	down1_text_attn Cross Attention	411K	4096 × 320	—
40	removed	to_feature_map Reshape		320 × 64 × 64	—
41	removed	downsample_1 Conv2d	6.4K	640 × 32 × 32	—
42	added	—			self_attn_1 Multi Head Attention	5.3M	256 × 1152
43	added	—			residual1_1 Add		256 × 1152
44	changed type, numGroups, numChannels, normalizedShape	mid_norm Group Norm	1.3K	640 × 32 × 32	norm2_1 Layer Norm	2.3K	256 × 1152
45	removed	to_tokens Reshape		1024 × 640	—
46	removed	mid_text_attn Cross Attention	1.6M	1024 × 640	—
47	removed	to_feature_map Reshape		640 × 32 × 32	—
48	removed	upsample_1 Upsample		640 × 64 × 64	—
49	removed	up1_conv Conv2d	3.2K	320 × 64 × 64	—
50	added	—			mlp_1 Feed Forward	11M	256 × 1152
51	added	—			residual2_1 Add		256 × 1152
52	changed type, numGroups, numChannels, normalizedShape	conv_out_norm Group Norm	640	320 × 64 × 64	norm1_2 Layer Norm	2.3K	256 × 1152
53	removed	up1_silu Swish		320 × 64 × 64	—
54	removed	conv_out Conv2d	40	4 × 64 × 64	—
55	added	—			self_attn_2 Multi Head Attention	5.3M	256 × 1152
56	added	—			residual1_2 Add		256 × 1152
57	added	—			norm2_2 Layer Norm	2.3K	256 × 1152
58	added	—			mlp_2 Feed Forward	11M	256 × 1152
59	added	—			residual2_2 Add		256 × 1152
60	added	—			norm1_3 Layer Norm	2.3K	256 × 1152
61	added	—			self_attn_3 Multi Head Attention	5.3M	256 × 1152
62	added	—			residual1_3 Add		256 × 1152
63	added	—			norm2_3 Layer Norm	2.3K	256 × 1152
64	added	—			mlp_3 Feed Forward	11M	256 × 1152
65	added	—			residual2_3 Add		256 × 1152
66	added	—			norm1_4 Layer Norm	2.3K	256 × 1152
67	added	—			self_attn_4 Multi Head Attention	5.3M	256 × 1152
68	added	—			residual1_4 Add		256 × 1152
69	added	—			norm2_4 Layer Norm	2.3K	256 × 1152
70	added	—			mlp_4 Feed Forward	11M	256 × 1152
71	added	—			residual2_4 Add		256 × 1152
72	added	—			norm1_5 Layer Norm	2.3K	256 × 1152
73	added	—			self_attn_5 Multi Head Attention	5.3M	256 × 1152
74	added	—			residual1_5 Add		256 × 1152
75	added	—			norm2_5 Layer Norm	2.3K	256 × 1152
76	added	—			mlp_5 Feed Forward	11M	256 × 1152
77	added	—			residual2_5 Add		256 × 1152
78	added	—			norm1_6 Layer Norm	2.3K	256 × 1152
79	added	—			self_attn_6 Multi Head Attention	5.3M	256 × 1152
80	added	—			residual1_6 Add		256 × 1152
81	added	—			norm2_6 Layer Norm	2.3K	256 × 1152
82	added	—			mlp_6 Feed Forward	11M	256 × 1152
83	added	—			residual2_6 Add		256 × 1152
84	added	—			norm1_7 Layer Norm	2.3K	256 × 1152
85	added	—			self_attn_7 Multi Head Attention	5.3M	256 × 1152
86	added	—			residual1_7 Add		256 × 1152
87	added	—			norm2_7 Layer Norm	2.3K	256 × 1152
88	added	—			mlp_7 Feed Forward	11M	256 × 1152
89	added	—			residual2_7 Add		256 × 1152
90	added	—			norm1_8 Layer Norm	2.3K	256 × 1152
91	added	—			self_attn_8 Multi Head Attention	5.3M	256 × 1152
92	added	—			residual1_8 Add		256 × 1152
93	added	—			norm2_8 Layer Norm	2.3K	256 × 1152
94	added	—			mlp_8 Feed Forward	11M	256 × 1152
95	added	—			residual2_8 Add		256 × 1152
96	added	—			norm1_9 Layer Norm	2.3K	256 × 1152
97	added	—			self_attn_9 Multi Head Attention	5.3M	256 × 1152
98	added	—			residual1_9 Add		256 × 1152
99	added	—			norm2_9 Layer Norm	2.3K	256 × 1152
100	added	—			mlp_9 Feed Forward	11M	256 × 1152
101	added	—			residual2_9 Add		256 × 1152
102	added	—			norm1_10 Layer Norm	2.3K	256 × 1152
103	added	—			self_attn_10 Multi Head Attention	5.3M	256 × 1152
104	added	—			residual1_10 Add		256 × 1152
105	added	—			norm2_10 Layer Norm	2.3K	256 × 1152
106	added	—			mlp_10 Feed Forward	11M	256 × 1152
107	added	—			residual2_10 Add		256 × 1152
108	added	—			norm1_11 Layer Norm	2.3K	256 × 1152
109	added	—			self_attn_11 Multi Head Attention	5.3M	256 × 1152
110	added	—			residual1_11 Add		256 × 1152
111	added	—			norm2_11 Layer Norm	2.3K	256 × 1152
112	added	—			mlp_11 Feed Forward	11M	256 × 1152
113	added	—			residual2_11 Add		256 × 1152
114	added	—			norm1_12 Layer Norm	2.3K	256 × 1152
115	added	—			self_attn_12 Multi Head Attention	5.3M	256 × 1152
116	added	—			residual1_12 Add		256 × 1152
117	added	—			norm2_12 Layer Norm	2.3K	256 × 1152
118	added	—			mlp_12 Feed Forward	11M	256 × 1152
119	added	—			residual2_12 Add		256 × 1152
120	added	—			norm1_13 Layer Norm	2.3K	256 × 1152
121	added	—			self_attn_13 Multi Head Attention	5.3M	256 × 1152
122	added	—			residual1_13 Add		256 × 1152
123	added	—			norm2_13 Layer Norm	2.3K	256 × 1152
124	added	—			mlp_13 Feed Forward	11M	256 × 1152
125	added	—			residual2_13 Add		256 × 1152
126	added	—			norm1_14 Layer Norm	2.3K	256 × 1152
127	added	—			self_attn_14 Multi Head Attention	5.3M	256 × 1152
128	added	—			residual1_14 Add		256 × 1152
129	added	—			norm2_14 Layer Norm	2.3K	256 × 1152
130	added	—			mlp_14 Feed Forward	11M	256 × 1152
131	added	—			residual2_14 Add		256 × 1152
132	added	—			norm1_15 Layer Norm	2.3K	256 × 1152
133	added	—			self_attn_15 Multi Head Attention	5.3M	256 × 1152
134	added	—			residual1_15 Add		256 × 1152
135	added	—			norm2_15 Layer Norm	2.3K	256 × 1152
136	added	—			mlp_15 Feed Forward	11M	256 × 1152
137	added	—			residual2_15 Add		256 × 1152
138	added	—			norm1_16 Layer Norm	2.3K	256 × 1152
139	added	—			self_attn_16 Multi Head Attention	5.3M	256 × 1152
140	added	—			residual1_16 Add		256 × 1152
141	added	—			norm2_16 Layer Norm	2.3K	256 × 1152
142	added	—			mlp_16 Feed Forward	11M	256 × 1152
143	added	—			residual2_16 Add		256 × 1152
144	added	—			norm1_17 Layer Norm	2.3K	256 × 1152
145	added	—			self_attn_17 Multi Head Attention	5.3M	256 × 1152
146	added	—			residual1_17 Add		256 × 1152
147	added	—			norm2_17 Layer Norm	2.3K	256 × 1152
148	added	—			mlp_17 Feed Forward	11M	256 × 1152
149	added	—			residual2_17 Add		256 × 1152
150	added	—			norm1_18 Layer Norm	2.3K	256 × 1152
151	added	—			self_attn_18 Multi Head Attention	5.3M	256 × 1152
152	added	—			residual1_18 Add		256 × 1152
153	added	—			norm2_18 Layer Norm	2.3K	256 × 1152
154	added	—			mlp_18 Feed Forward	11M	256 × 1152
155	added	—			residual2_18 Add		256 × 1152
156	added	—			norm1_19 Layer Norm	2.3K	256 × 1152
157	added	—			self_attn_19 Multi Head Attention	5.3M	256 × 1152
158	added	—			residual1_19 Add		256 × 1152
159	added	—			norm2_19 Layer Norm	2.3K	256 × 1152
160	added	—			mlp_19 Feed Forward	11M	256 × 1152
161	added	—			residual2_19 Add		256 × 1152
162	added	—			norm1_20 Layer Norm	2.3K	256 × 1152
163	added	—			self_attn_20 Multi Head Attention	5.3M	256 × 1152
164	added	—			residual1_20 Add		256 × 1152
165	added	—			norm2_20 Layer Norm	2.3K	256 × 1152
166	added	—			mlp_20 Feed Forward	11M	256 × 1152
167	added	—			residual2_20 Add		256 × 1152
168	added	—			norm1_21 Layer Norm	2.3K	256 × 1152
169	added	—			self_attn_21 Multi Head Attention	5.3M	256 × 1152
170	added	—			residual1_21 Add		256 × 1152
171	added	—			norm2_21 Layer Norm	2.3K	256 × 1152
172	added	—			mlp_21 Feed Forward	11M	256 × 1152
173	added	—			residual2_21 Add		256 × 1152
174	added	—			norm1_22 Layer Norm	2.3K	256 × 1152
175	added	—			self_attn_22 Multi Head Attention	5.3M	256 × 1152
176	added	—			residual1_22 Add		256 × 1152
177	added	—			norm2_22 Layer Norm	2.3K	256 × 1152
178	added	—			mlp_22 Feed Forward	11M	256 × 1152
179	added	—			residual2_22 Add		256 × 1152
180	added	—			norm1_23 Layer Norm	2.3K	256 × 1152
181	added	—			self_attn_23 Multi Head Attention	5.3M	256 × 1152
182	added	—			residual1_23 Add		256 × 1152
183	added	—			norm2_23 Layer Norm	2.3K	256 × 1152
184	added	—			mlp_23 Feed Forward	11M	256 × 1152
185	added	—			residual2_23 Add		256 × 1152
186	added	—			norm1_24 Layer Norm	2.3K	256 × 1152
187	added	—			self_attn_24 Multi Head Attention	5.3M	256 × 1152
188	added	—			residual1_24 Add		256 × 1152
189	added	—			norm2_24 Layer Norm	2.3K	256 × 1152
190	added	—			mlp_24 Feed Forward	11M	256 × 1152
191	added	—			residual2_24 Add		256 × 1152
192	added	—			norm1_25 Layer Norm	2.3K	256 × 1152
193	added	—			self_attn_25 Multi Head Attention	5.3M	256 × 1152
194	added	—			residual1_25 Add		256 × 1152
195	added	—			norm2_25 Layer Norm	2.3K	256 × 1152
196	added	—			mlp_25 Feed Forward	11M	256 × 1152
197	added	—			residual2_25 Add		256 × 1152
198	added	—			norm1_26 Layer Norm	2.3K	256 × 1152
199	added	—			self_attn_26 Multi Head Attention	5.3M	256 × 1152
200	added	—			residual1_26 Add		256 × 1152
201	added	—			norm2_26 Layer Norm	2.3K	256 × 1152
202	added	—			mlp_26 Feed Forward	11M	256 × 1152
203	added	—			residual2_26 Add		256 × 1152
204	added	—			norm1_27 Layer Norm	2.3K	256 × 1152
205	added	—			self_attn_27 Multi Head Attention	5.3M	256 × 1152
206	added	—			residual1_27 Add		256 × 1152
207	added	—			norm2_27 Layer Norm	2.3K	256 × 1152
208	added	—			mlp_27 Feed Forward	11M	256 × 1152
209	added	—			residual2_27 Add		256 × 1152
210	added	—			norm1_28 Layer Norm	2.3K	256 × 1152
211	added	—			self_attn_28 Multi Head Attention	5.3M	256 × 1152
212	added	—			residual1_28 Add		256 × 1152
213	added	—			norm2_28 Layer Norm	2.3K	256 × 1152
214	added	—			mlp_28 Feed Forward	11M	256 × 1152
215	added	—			residual2_28 Add		256 × 1152
216	added	—			final_norm Layer Norm	2.3K	256 × 1152
217	added	—			unpatchify Linear	37K	256 × 32
218	same	predicted_noise Output		4 × 64 × 64	predicted_noise Output		256 × 32

What this is not

The two are priced at different declared inputs (4 × 64 × 64 against 4 × 32 × 32), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

Take it further

Open either graph in the editor, change it, and check it again: Diffusion UNet · DiT-XL/2

Compare any two models of your own, including anything on Hugging Face: the comparison tool.

Machine-readable: this page as markdown · the pair index · POST https://www.neurarch.com/api/v1/plan for a graph of your own.

Related comparisons

U-Net vs Diffusion UNet

A segmentation U-Net against the one a diffusion model uses.

ViT-B/16 vs DiT-XL/2

A vision transformer against a diffusion transformer.