# ViT-B/16 vs Swin-Tiny

A flat vision transformer against a hierarchical one.

**Swin-Tiny has 20M more parameters than ViT-B/16: 74 layers added, 2 removed, 4 changed.**

Source: https://neurarch.com/diff/vit-b16-vs-swin-tiny.html

## Sides

| | ViT-B/16 | Swin-Tiny |
|---|---|---|
| Layers | 11 | 83 |
| Parameters | 8.4M | 28M |
| Input | 3 × 224 × 224 | 3 × 224 × 224 |
| Output | 196 × 1000 | 1000 |
| Forward-passes | yes | yes |
| Est. train cost | $0.105 | $0.288 |
| T4 16GB | fits | fits |
| A100 40GB | fits | fits |
| H100 80GB | fits | fits |

## Deltas (Swin-Tiny relative to ViT-B/16)

- Parameters: +20M (+234%)
- Layers: +72
- Added 74, removed 2, changed 4, unchanged 7

## Layer by layer

| # | Status | ViT-B/16 | Params | Output | Swin-Tiny | Params | Output |
|---|---|---|---|---|---|---|---|
| 1 | same | image (Input) |  | 3 × 224 × 224 | image (Input) |  | 3 × 224 × 224 |
| 2 | changed (patchSize, embedDim) | patch_embed (Patch Embed) | 591K | 196 × 768 | patch_embed_4x4 (Patch Embed) | 4.7K | 3136 × 96 |
| 3 | removed | pos_embed (Positional Encoding) |  | 196 × 768 |  | |  |
| 4 | removed | dropout (Dropout) |  | 196 × 768 |  | |  |
| 5 | added |  | |  | s1b1_norm1 (Layer Norm) | 192 | 3136 × 96 |
| 6 | added |  | |  | s1b1_window_attn (Multi Head Attention) | 37K | 3136 × 96 |
| 7 | added |  | |  | s1b1_res1 (Add) |  | 3136 × 96 |
| 8 | added |  | |  | s1b1_norm2 (Layer Norm) | 192 | 3136 × 96 |
| 9 | added |  | |  | s1b1_mlp (Feed Forward) | 74K | 3136 × 96 |
| 10 | added |  | |  | s1b1_res2 (Add) |  | 3136 × 96 |
| 11 | added |  | |  | s1b2_norm1 (Layer Norm) | 192 | 3136 × 96 |
| 12 | added |  | |  | s1b2_shifted_window_attn (Multi Head Attention) | 37K | 3136 × 96 |
| 13 | added |  | |  | s1b2_res1 (Add) |  | 3136 × 96 |
| 14 | added |  | |  | s1b2_norm2 (Layer Norm) | 192 | 3136 × 96 |
| 15 | added |  | |  | s1b2_mlp (Feed Forward) | 74K | 3136 × 96 |
| 16 | added |  | |  | s1b2_res2 (Add) |  | 3136 × 96 |
| 17 | added |  | |  | merge_patches_n15 (Reshape) |  | 784 × 384 |
| 18 | added |  | |  | patch_merging_2 (Linear) | 74K | 784 × 192 |
| 19 | added |  | |  | s2b1_norm1 (Layer Norm) | 384 | 784 × 192 |
| 20 | added |  | |  | s2b1_window_attn (Multi Head Attention) | 148K | 784 × 192 |
| 21 | added |  | |  | s2b1_res1 (Add) |  | 784 × 192 |
| 22 | added |  | |  | s2b1_norm2 (Layer Norm) | 384 | 784 × 192 |
| 23 | added |  | |  | s2b1_mlp (Feed Forward) | 296K | 784 × 192 |
| 24 | added |  | |  | s2b1_res2 (Add) |  | 784 × 192 |
| 25 | added |  | |  | s2b2_norm1 (Layer Norm) | 384 | 784 × 192 |
| 26 | added |  | |  | s2b2_shifted_window_attn (Multi Head Attention) | 148K | 784 × 192 |
| 27 | added |  | |  | s2b2_res1 (Add) |  | 784 × 192 |
| 28 | added |  | |  | s2b2_norm2 (Layer Norm) | 384 | 784 × 192 |
| 29 | added |  | |  | s2b2_mlp (Feed Forward) | 296K | 784 × 192 |
| 30 | added |  | |  | s2b2_res2 (Add) |  | 784 × 192 |
| 31 | added |  | |  | merge_patches_n28 (Reshape) |  | 196 × 768 |
| 32 | added |  | |  | patch_merging_3 (Linear) | 295K | 196 × 384 |
| 33 | added |  | |  | s3b1_norm1 (Layer Norm) | 768 | 196 × 384 |
| 34 | added |  | |  | s3b1_window_attn (Multi Head Attention) | 591K | 196 × 384 |
| 35 | added |  | |  | s3b1_res1 (Add) |  | 196 × 384 |
| 36 | added |  | |  | s3b1_norm2 (Layer Norm) | 768 | 196 × 384 |
| 37 | added |  | |  | s3b1_mlp (Feed Forward) | 1.2M | 196 × 384 |
| 38 | added |  | |  | s3b1_res2 (Add) |  | 196 × 384 |
| 39 | added |  | |  | s3b2_norm1 (Layer Norm) | 768 | 196 × 384 |
| 40 | added |  | |  | s3b2_shifted_window_attn (Multi Head Attention) | 591K | 196 × 384 |
| 41 | added |  | |  | s3b2_res1 (Add) |  | 196 × 384 |
| 42 | added |  | |  | s3b2_norm2 (Layer Norm) | 768 | 196 × 384 |
| 43 | added |  | |  | s3b2_mlp (Feed Forward) | 1.2M | 196 × 384 |
| 44 | added |  | |  | s3b2_res2 (Add) |  | 196 × 384 |
| 45 | added |  | |  | s3b3_norm1 (Layer Norm) | 768 | 196 × 384 |
| 46 | added |  | |  | s3b3_window_attn (Multi Head Attention) | 591K | 196 × 384 |
| 47 | added |  | |  | s3b3_res1 (Add) |  | 196 × 384 |
| 48 | added |  | |  | s3b3_norm2 (Layer Norm) | 768 | 196 × 384 |
| 49 | added |  | |  | s3b3_mlp (Feed Forward) | 1.2M | 196 × 384 |
| 50 | added |  | |  | s3b3_res2 (Add) |  | 196 × 384 |
| 51 | added |  | |  | s3b4_norm1 (Layer Norm) | 768 | 196 × 384 |
| 52 | added |  | |  | s3b4_shifted_window_attn (Multi Head Attention) | 591K | 196 × 384 |
| 53 | added |  | |  | s3b4_res1 (Add) |  | 196 × 384 |
| 54 | added |  | |  | s3b4_norm2 (Layer Norm) | 768 | 196 × 384 |
| 55 | added |  | |  | s3b4_mlp (Feed Forward) | 1.2M | 196 × 384 |
| 56 | added |  | |  | s3b4_res2 (Add) |  | 196 × 384 |
| 57 | added |  | |  | s3b5_norm1 (Layer Norm) | 768 | 196 × 384 |
| 58 | added |  | |  | s3b5_window_attn (Multi Head Attention) | 591K | 196 × 384 |
| 59 | added |  | |  | s3b5_res1 (Add) |  | 196 × 384 |
| 60 | added |  | |  | s3b5_norm2 (Layer Norm) | 768 | 196 × 384 |
| 61 | added |  | |  | s3b5_mlp (Feed Forward) | 1.2M | 196 × 384 |
| 62 | added |  | |  | s3b5_res2 (Add) |  | 196 × 384 |
| 63 | added |  | |  | s3b6_norm1 (Layer Norm) | 768 | 196 × 384 |
| 64 | added |  | |  | s3b6_shifted_window_attn (Multi Head Attention) | 591K | 196 × 384 |
| 65 | added |  | |  | s3b6_res1 (Add) |  | 196 × 384 |
| 66 | added |  | |  | s3b6_norm2 (Layer Norm) | 768 | 196 × 384 |
| 67 | added |  | |  | s3b6_mlp (Feed Forward) | 1.2M | 196 × 384 |
| 68 | added |  | |  | s3b6_res2 (Add) |  | 196 × 384 |
| 69 | added |  | |  | merge_patches_n65 (Reshape) |  | 49 × 1536 |
| 70 | added |  | |  | patch_merging_4 (Linear) | 1.2M | 49 × 768 |
| 71 | added |  | |  | s4b1_norm1 (Layer Norm) | 1.5K | 49 × 768 |
| 72 | added |  | |  | s4b1_window_attn (Multi Head Attention) | 2.4M | 49 × 768 |
| 73 | added |  | |  | s4b1_res1 (Add) |  | 49 × 768 |
| 74 | added |  | |  | s4b1_norm2 (Layer Norm) | 1.5K | 49 × 768 |
| 75 | added |  | |  | s4b1_mlp (Feed Forward) | 4.7M | 49 × 768 |
| 76 | added |  | |  | s4b1_res2 (Add) |  | 49 × 768 |
| 77 | same | norm_1 (Layer Norm) | 1.5K | 196 × 768 | s4b2_norm1 (Layer Norm) | 1.5K | 49 × 768 |
| 78 | changed (numHeads) | attn (Multi Head Attention) | 2.4M | 196 × 768 | s4b2_shifted_window_attn (Multi Head Attention) | 2.4M | 49 × 768 |
| 79 | same | residual_1 (Add) |  | 196 × 768 | s4b2_res1 (Add) |  | 49 × 768 |
| 80 | same | norm_2 (Layer Norm) | 1.5K | 196 × 768 | s4b2_norm2 (Layer Norm) | 1.5K | 49 × 768 |
| 81 | changed (hiddenDim, embedDim) | mlp (Feed Forward) | 4.7M | 196 × 768 | s4b2_mlp (Feed Forward) | 4.7M | 49 × 768 |
| 82 | same | residual_2 (Add) |  | 196 × 768 | s4b2_res2 (Add) |  | 49 × 768 |
| 83 | same | norm_final (Layer Norm) | 1.5K | 196 × 768 | final_norm (Layer Norm) | 1.5K | 49 × 768 |
| 84 | added |  | |  | to_channels (Permute) |  | 768 × 49 |
| 85 | added |  | |  | avgpool (Global Avg Pool1d) |  | 768 |
| 86 | changed (inFeatures) | head (Linear) |  | 196 × 1000 | classifier (Linear) | 769K | 1000 |
| 87 | same | class_logits (Output) |  | 196 × 1000 | class_logits (Output) |  | 1000 |

## What this is not

- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

## Graphs

- ViT-B/16: https://neurarch.com/templates/vit-b16/model.json
- Swin-Tiny: https://neurarch.com/templates/swin-tiny/model.json
- Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`