# Diffusion UNet vs DiT-XL/2 Convolutional against transformer backbones for diffusion. **DiT-XL/2 has 664M more parameters than Diffusion UNet: 199 layers added, 14 removed, 4 changed.** Source: https://neurarch.com/diff/diffusion-unet-vs-dit-xl2.html ## Sides | | Diffusion UNet | DiT-XL/2 | |---|---|---| | Layers | 17 | 201 | | Parameters | 6.7M | 671M | | Input | 4 × 64 × 64 | 4 × 32 × 32 | | Output | 4 × 64 × 64 | 256 × 32 | | Forward-passes | yes | yes | | Est. train cost | $0.541 | $4.41 | | T4 16GB | fits | fits | | A100 40GB | fits | fits | | H100 80GB | fits | fits | ## Deltas (DiT-XL/2 relative to Diffusion UNet) - Parameters: +664M (100× the size) - Layers: +184 - Added 199, removed 14, changed 4, unchanged 1 ## Layer by layer | # | Status | Diffusion UNet | Params | Output | DiT-XL/2 | Params | Output | |---|---|---|---|---|---|---|---| | 1 | changed (shape) | noisy_latent (Input) | | 4 × 64 × 64 | noisy_latent (Input) | | 4 × 32 × 32 | | 2 | removed | conv_in (Conv2d) | 3.2K | 320 × 64 × 64 | | | | | 3 | added | | | | timestep_+_class (Input) | | 1 × 2 | | 4 | added | | | | patchify_2x2 (Patch Embed) | 20K | 256 × 1152 | | 5 | added | | | | cond_embed (Embedding) | 1.2M | 1 × 2 × 1152 | | 6 | added | | | | pos_embed (Positional Encoding) | | 256 × 1152 | | 7 | added | | | | adaLN_1 (Linear) | 8.0M | 1 × 2 × 6912 | | 8 | added | | | | adaLN_2 (Linear) | 8.0M | 1 × 2 × 6912 | | 9 | added | | | | adaLN_3 (Linear) | 8.0M | 1 × 2 × 6912 | | 10 | added | | | | adaLN_4 (Linear) | 8.0M | 1 × 2 × 6912 | | 11 | added | | | | adaLN_5 (Linear) | 8.0M | 1 × 2 × 6912 | | 12 | added | | | | adaLN_6 (Linear) | 8.0M | 1 × 2 × 6912 | | 13 | added | | | | adaLN_7 (Linear) | 8.0M | 1 × 2 × 6912 | | 14 | added | | | | adaLN_8 (Linear) | 8.0M | 1 × 2 × 6912 | | 15 | added | | | | adaLN_9 (Linear) | 8.0M | 1 × 2 × 6912 | | 16 | added | | | | adaLN_10 (Linear) | 8.0M | 1 × 2 × 6912 | | 17 | added | | | | adaLN_11 (Linear) | 8.0M | 1 × 2 × 6912 | | 18 | added | | | | adaLN_12 (Linear) | 8.0M | 1 × 2 × 6912 | | 19 | added | | | | adaLN_13 (Linear) | 8.0M | 1 × 2 × 6912 | | 20 | added | | | | adaLN_14 (Linear) | 8.0M | 1 × 2 × 6912 | | 21 | added | | | | adaLN_15 (Linear) | 8.0M | 1 × 2 × 6912 | | 22 | added | | | | adaLN_16 (Linear) | 8.0M | 1 × 2 × 6912 | | 23 | added | | | | adaLN_17 (Linear) | 8.0M | 1 × 2 × 6912 | | 24 | added | | | | adaLN_18 (Linear) | 8.0M | 1 × 2 × 6912 | | 25 | added | | | | adaLN_19 (Linear) | 8.0M | 1 × 2 × 6912 | | 26 | added | | | | adaLN_20 (Linear) | 8.0M | 1 × 2 × 6912 | | 27 | added | | | | adaLN_21 (Linear) | 8.0M | 1 × 2 × 6912 | | 28 | added | | | | adaLN_22 (Linear) | 8.0M | 1 × 2 × 6912 | | 29 | added | | | | adaLN_23 (Linear) | 8.0M | 1 × 2 × 6912 | | 30 | added | | | | adaLN_24 (Linear) | 8.0M | 1 × 2 × 6912 | | 31 | added | | | | adaLN_25 (Linear) | 8.0M | 1 × 2 × 6912 | | 32 | added | | | | adaLN_26 (Linear) | 8.0M | 1 × 2 × 6912 | | 33 | added | | | | adaLN_27 (Linear) | 8.0M | 1 × 2 × 6912 | | 34 | added | | | | adaLN_28 (Linear) | 8.0M | 1 × 2 × 6912 | | 35 | changed (type, numGroups, numChannels, normalizedShape) | down1_norm (Group Norm) | 640 | 320 × 64 × 64 | norm1_1 (Layer Norm) | 2.3K | 256 × 1152 | | 36 | removed | down1_conv (Conv2d) | 3.2K | 320 × 64 × 64 | | | | | 37 | removed | down1_silu (Swish) | | 320 × 64 × 64 | | | | | 38 | removed | to_tokens (Reshape) | | 4096 × 320 | | | | | 39 | removed | down1_text_attn (Cross Attention) | 411K | 4096 × 320 | | | | | 40 | removed | to_feature_map (Reshape) | | 320 × 64 × 64 | | | | | 41 | removed | downsample_1 (Conv2d) | 6.4K | 640 × 32 × 32 | | | | | 42 | added | | | | self_attn_1 (Multi Head Attention) | 5.3M | 256 × 1152 | | 43 | added | | | | residual1_1 (Add) | | 256 × 1152 | | 44 | changed (type, numGroups, numChannels, normalizedShape) | mid_norm (Group Norm) | 1.3K | 640 × 32 × 32 | norm2_1 (Layer Norm) | 2.3K | 256 × 1152 | | 45 | removed | to_tokens (Reshape) | | 1024 × 640 | | | | | 46 | removed | mid_text_attn (Cross Attention) | 1.6M | 1024 × 640 | | | | | 47 | removed | to_feature_map (Reshape) | | 640 × 32 × 32 | | | | | 48 | removed | upsample_1 (Upsample) | | 640 × 64 × 64 | | | | | 49 | removed | up1_conv (Conv2d) | 3.2K | 320 × 64 × 64 | | | | | 50 | added | | | | mlp_1 (Feed Forward) | 11M | 256 × 1152 | | 51 | added | | | | residual2_1 (Add) | | 256 × 1152 | | 52 | changed (type, numGroups, numChannels, normalizedShape) | conv_out_norm (Group Norm) | 640 | 320 × 64 × 64 | norm1_2 (Layer Norm) | 2.3K | 256 × 1152 | | 53 | removed | up1_silu (Swish) | | 320 × 64 × 64 | | | | | 54 | removed | conv_out (Conv2d) | 40 | 4 × 64 × 64 | | | | | 55 | added | | | | self_attn_2 (Multi Head Attention) | 5.3M | 256 × 1152 | | 56 | added | | | | residual1_2 (Add) | | 256 × 1152 | | 57 | added | | | | norm2_2 (Layer Norm) | 2.3K | 256 × 1152 | | 58 | added | | | | mlp_2 (Feed Forward) | 11M | 256 × 1152 | | 59 | added | | | | residual2_2 (Add) | | 256 × 1152 | | 60 | added | | | | norm1_3 (Layer Norm) | 2.3K | 256 × 1152 | | 61 | added | | | | self_attn_3 (Multi Head Attention) | 5.3M | 256 × 1152 | | 62 | added | | | | residual1_3 (Add) | | 256 × 1152 | | 63 | added | | | | norm2_3 (Layer Norm) | 2.3K | 256 × 1152 | | 64 | added | | | | mlp_3 (Feed Forward) | 11M | 256 × 1152 | | 65 | added | | | | residual2_3 (Add) | | 256 × 1152 | | 66 | added | | | | norm1_4 (Layer Norm) | 2.3K | 256 × 1152 | | 67 | added | | | | self_attn_4 (Multi Head Attention) | 5.3M | 256 × 1152 | | 68 | added | | | | residual1_4 (Add) | | 256 × 1152 | | 69 | added | | | | norm2_4 (Layer Norm) | 2.3K | 256 × 1152 | | 70 | added | | | | mlp_4 (Feed Forward) | 11M | 256 × 1152 | | 71 | added | | | | residual2_4 (Add) | | 256 × 1152 | | 72 | added | | | | norm1_5 (Layer Norm) | 2.3K | 256 × 1152 | | 73 | added | | | | self_attn_5 (Multi Head Attention) | 5.3M | 256 × 1152 | | 74 | added | | | | residual1_5 (Add) | | 256 × 1152 | | 75 | added | | | | norm2_5 (Layer Norm) | 2.3K | 256 × 1152 | | 76 | added | | | | mlp_5 (Feed Forward) | 11M | 256 × 1152 | | 77 | added | | | | residual2_5 (Add) | | 256 × 1152 | | 78 | added | | | | norm1_6 (Layer Norm) | 2.3K | 256 × 1152 | | 79 | added | | | | self_attn_6 (Multi Head Attention) | 5.3M | 256 × 1152 | | 80 | added | | | | residual1_6 (Add) | | 256 × 1152 | | 81 | added | | | | norm2_6 (Layer Norm) | 2.3K | 256 × 1152 | | 82 | added | | | | mlp_6 (Feed Forward) | 11M | 256 × 1152 | | 83 | added | | | | residual2_6 (Add) | | 256 × 1152 | | 84 | added | | | | norm1_7 (Layer Norm) | 2.3K | 256 × 1152 | | 85 | added | | | | self_attn_7 (Multi Head Attention) | 5.3M | 256 × 1152 | | 86 | added | | | | residual1_7 (Add) | | 256 × 1152 | | 87 | added | | | | norm2_7 (Layer Norm) | 2.3K | 256 × 1152 | | 88 | added | | | | mlp_7 (Feed Forward) | 11M | 256 × 1152 | | 89 | added | | | | residual2_7 (Add) | | 256 × 1152 | | 90 | added | | | | norm1_8 (Layer Norm) | 2.3K | 256 × 1152 | | 91 | added | | | | self_attn_8 (Multi Head Attention) | 5.3M | 256 × 1152 | | 92 | added | | | | residual1_8 (Add) | | 256 × 1152 | | 93 | added | | | | norm2_8 (Layer Norm) | 2.3K | 256 × 1152 | | 94 | added | | | | mlp_8 (Feed Forward) | 11M | 256 × 1152 | | 95 | added | | | | residual2_8 (Add) | | 256 × 1152 | | 96 | added | | | | norm1_9 (Layer Norm) | 2.3K | 256 × 1152 | | 97 | added | | | | self_attn_9 (Multi Head Attention) | 5.3M | 256 × 1152 | | 98 | added | | | | residual1_9 (Add) | | 256 × 1152 | | 99 | added | | | | norm2_9 (Layer Norm) | 2.3K | 256 × 1152 | | 100 | added | | | | mlp_9 (Feed Forward) | 11M | 256 × 1152 | | 101 | added | | | | residual2_9 (Add) | | 256 × 1152 | | 102 | added | | | | norm1_10 (Layer Norm) | 2.3K | 256 × 1152 | | 103 | added | | | | self_attn_10 (Multi Head Attention) | 5.3M | 256 × 1152 | | 104 | added | | | | residual1_10 (Add) | | 256 × 1152 | | 105 | added | | | | norm2_10 (Layer Norm) | 2.3K | 256 × 1152 | | 106 | added | | | | mlp_10 (Feed Forward) | 11M | 256 × 1152 | | 107 | added | | | | residual2_10 (Add) | | 256 × 1152 | | 108 | added | | | | norm1_11 (Layer Norm) | 2.3K | 256 × 1152 | | 109 | added | | | | self_attn_11 (Multi Head Attention) | 5.3M | 256 × 1152 | | 110 | added | | | | residual1_11 (Add) | | 256 × 1152 | | 111 | added | | | | norm2_11 (Layer Norm) | 2.3K | 256 × 1152 | | 112 | added | | | | mlp_11 (Feed Forward) | 11M | 256 × 1152 | | 113 | added | | | | residual2_11 (Add) | | 256 × 1152 | | 114 | added | | | | norm1_12 (Layer Norm) | 2.3K | 256 × 1152 | | 115 | added | | | | self_attn_12 (Multi Head Attention) | 5.3M | 256 × 1152 | | 116 | added | | | | residual1_12 (Add) | | 256 × 1152 | | 117 | added | | | | norm2_12 (Layer Norm) | 2.3K | 256 × 1152 | | 118 | added | | | | mlp_12 (Feed Forward) | 11M | 256 × 1152 | | 119 | added | | | | residual2_12 (Add) | | 256 × 1152 | | 120 | added | | | | norm1_13 (Layer Norm) | 2.3K | 256 × 1152 | | 121 | added | | | | self_attn_13 (Multi Head Attention) | 5.3M | 256 × 1152 | | 122 | added | | | | residual1_13 (Add) | | 256 × 1152 | | 123 | added | | | | norm2_13 (Layer Norm) | 2.3K | 256 × 1152 | | 124 | added | | | | mlp_13 (Feed Forward) | 11M | 256 × 1152 | | 125 | added | | | | residual2_13 (Add) | | 256 × 1152 | | 126 | added | | | | norm1_14 (Layer Norm) | 2.3K | 256 × 1152 | | 127 | added | | | | self_attn_14 (Multi Head Attention) | 5.3M | 256 × 1152 | | 128 | added | | | | residual1_14 (Add) | | 256 × 1152 | | 129 | added | | | | norm2_14 (Layer Norm) | 2.3K | 256 × 1152 | | 130 | added | | | | mlp_14 (Feed Forward) | 11M | 256 × 1152 | | 131 | added | | | | residual2_14 (Add) | | 256 × 1152 | | 132 | added | | | | norm1_15 (Layer Norm) | 2.3K | 256 × 1152 | | 133 | added | | | | self_attn_15 (Multi Head Attention) | 5.3M | 256 × 1152 | | 134 | added | | | | residual1_15 (Add) | | 256 × 1152 | | 135 | added | | | | norm2_15 (Layer Norm) | 2.3K | 256 × 1152 | | 136 | added | | | | mlp_15 (Feed Forward) | 11M | 256 × 1152 | | 137 | added | | | | residual2_15 (Add) | | 256 × 1152 | | 138 | added | | | | norm1_16 (Layer Norm) | 2.3K | 256 × 1152 | | 139 | added | | | | self_attn_16 (Multi Head Attention) | 5.3M | 256 × 1152 | | 140 | added | | | | residual1_16 (Add) | | 256 × 1152 | | 141 | added | | | | norm2_16 (Layer Norm) | 2.3K | 256 × 1152 | | 142 | added | | | | mlp_16 (Feed Forward) | 11M | 256 × 1152 | | 143 | added | | | | residual2_16 (Add) | | 256 × 1152 | | 144 | added | | | | norm1_17 (Layer Norm) | 2.3K | 256 × 1152 | | 145 | added | | | | self_attn_17 (Multi Head Attention) | 5.3M | 256 × 1152 | | 146 | added | | | | residual1_17 (Add) | | 256 × 1152 | | 147 | added | | | | norm2_17 (Layer Norm) | 2.3K | 256 × 1152 | | 148 | added | | | | mlp_17 (Feed Forward) | 11M | 256 × 1152 | | 149 | added | | | | residual2_17 (Add) | | 256 × 1152 | | 150 | added | | | | norm1_18 (Layer Norm) | 2.3K | 256 × 1152 | | 151 | added | | | | self_attn_18 (Multi Head Attention) | 5.3M | 256 × 1152 | | 152 | added | | | | residual1_18 (Add) | | 256 × 1152 | | 153 | added | | | | norm2_18 (Layer Norm) | 2.3K | 256 × 1152 | | 154 | added | | | | mlp_18 (Feed Forward) | 11M | 256 × 1152 | | 155 | added | | | | residual2_18 (Add) | | 256 × 1152 | | 156 | added | | | | norm1_19 (Layer Norm) | 2.3K | 256 × 1152 | | 157 | added | | | | self_attn_19 (Multi Head Attention) | 5.3M | 256 × 1152 | | 158 | added | | | | residual1_19 (Add) | | 256 × 1152 | | 159 | added | | | | norm2_19 (Layer Norm) | 2.3K | 256 × 1152 | | 160 | added | | | | mlp_19 (Feed Forward) | 11M | 256 × 1152 | | 161 | added | | | | residual2_19 (Add) | | 256 × 1152 | | 162 | added | | | | norm1_20 (Layer Norm) | 2.3K | 256 × 1152 | | 163 | added | | | | self_attn_20 (Multi Head Attention) | 5.3M | 256 × 1152 | | 164 | added | | | | residual1_20 (Add) | | 256 × 1152 | | 165 | added | | | | norm2_20 (Layer Norm) | 2.3K | 256 × 1152 | | 166 | added | | | | mlp_20 (Feed Forward) | 11M | 256 × 1152 | | 167 | added | | | | residual2_20 (Add) | | 256 × 1152 | | 168 | added | | | | norm1_21 (Layer Norm) | 2.3K | 256 × 1152 | | 169 | added | | | | self_attn_21 (Multi Head Attention) | 5.3M | 256 × 1152 | | 170 | added | | | | residual1_21 (Add) | | 256 × 1152 | | 171 | added | | | | norm2_21 (Layer Norm) | 2.3K | 256 × 1152 | | 172 | added | | | | mlp_21 (Feed Forward) | 11M | 256 × 1152 | | 173 | added | | | | residual2_21 (Add) | | 256 × 1152 | | 174 | added | | | | norm1_22 (Layer Norm) | 2.3K | 256 × 1152 | | 175 | added | | | | self_attn_22 (Multi Head Attention) | 5.3M | 256 × 1152 | | 176 | added | | | | residual1_22 (Add) | | 256 × 1152 | | 177 | added | | | | norm2_22 (Layer Norm) | 2.3K | 256 × 1152 | | 178 | added | | | | mlp_22 (Feed Forward) | 11M | 256 × 1152 | | 179 | added | | | | residual2_22 (Add) | | 256 × 1152 | | 180 | added | | | | norm1_23 (Layer Norm) | 2.3K | 256 × 1152 | | 181 | added | | | | self_attn_23 (Multi Head Attention) | 5.3M | 256 × 1152 | | 182 | added | | | | residual1_23 (Add) | | 256 × 1152 | | 183 | added | | | | norm2_23 (Layer Norm) | 2.3K | 256 × 1152 | | 184 | added | | | | mlp_23 (Feed Forward) | 11M | 256 × 1152 | | 185 | added | | | | residual2_23 (Add) | | 256 × 1152 | | 186 | added | | | | norm1_24 (Layer Norm) | 2.3K | 256 × 1152 | | 187 | added | | | | self_attn_24 (Multi Head Attention) | 5.3M | 256 × 1152 | | 188 | added | | | | residual1_24 (Add) | | 256 × 1152 | | 189 | added | | | | norm2_24 (Layer Norm) | 2.3K | 256 × 1152 | | 190 | added | | | | mlp_24 (Feed Forward) | 11M | 256 × 1152 | | 191 | added | | | | residual2_24 (Add) | | 256 × 1152 | | 192 | added | | | | norm1_25 (Layer Norm) | 2.3K | 256 × 1152 | | 193 | added | | | | self_attn_25 (Multi Head Attention) | 5.3M | 256 × 1152 | | 194 | added | | | | residual1_25 (Add) | | 256 × 1152 | | 195 | added | | | | norm2_25 (Layer Norm) | 2.3K | 256 × 1152 | | 196 | added | | | | mlp_25 (Feed Forward) | 11M | 256 × 1152 | | 197 | added | | | | residual2_25 (Add) | | 256 × 1152 | | 198 | added | | | | norm1_26 (Layer Norm) | 2.3K | 256 × 1152 | | 199 | added | | | | self_attn_26 (Multi Head Attention) | 5.3M | 256 × 1152 | | 200 | added | | | | residual1_26 (Add) | | 256 × 1152 | | 201 | added | | | | norm2_26 (Layer Norm) | 2.3K | 256 × 1152 | | 202 | added | | | | mlp_26 (Feed Forward) | 11M | 256 × 1152 | | 203 | added | | | | residual2_26 (Add) | | 256 × 1152 | | 204 | added | | | | norm1_27 (Layer Norm) | 2.3K | 256 × 1152 | | 205 | added | | | | self_attn_27 (Multi Head Attention) | 5.3M | 256 × 1152 | | 206 | added | | | | residual1_27 (Add) | | 256 × 1152 | | 207 | added | | | | norm2_27 (Layer Norm) | 2.3K | 256 × 1152 | | 208 | added | | | | mlp_27 (Feed Forward) | 11M | 256 × 1152 | | 209 | added | | | | residual2_27 (Add) | | 256 × 1152 | | 210 | added | | | | norm1_28 (Layer Norm) | 2.3K | 256 × 1152 | | 211 | added | | | | self_attn_28 (Multi Head Attention) | 5.3M | 256 × 1152 | | 212 | added | | | | residual1_28 (Add) | | 256 × 1152 | | 213 | added | | | | norm2_28 (Layer Norm) | 2.3K | 256 × 1152 | | 214 | added | | | | mlp_28 (Feed Forward) | 11M | 256 × 1152 | | 215 | added | | | | residual2_28 (Add) | | 256 × 1152 | | 216 | added | | | | final_norm (Layer Norm) | 2.3K | 256 × 1152 | | 217 | added | | | | unpatchify (Linear) | 37K | 256 × 32 | | 218 | same | predicted_noise (Output) | | 4 × 64 × 64 | predicted_noise (Output) | | 256 × 32 | ## What this is not - The two are priced at different declared inputs (4 × 64 × 64 against 4 × 32 × 32), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected. - Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not. - Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run. ## Graphs - Diffusion UNet: https://neurarch.com/templates/diffusion-unet/model.json - DiT-XL/2: https://neurarch.com/templates/dit-xl2/model.json - Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`