# Swin-Tiny

> Hierarchical vision transformer — shifted-window attention builds a feature pyramid for dense prediction (Liu 2021)

Pick when you need a ViT that produces multi-scale features (detection, segmentation) rather than a single-scale ViT. Windowed attention keeps compute linear in image size.

- Category: Computer Vision
- Layers: 81
- Parameters: 28.26M
- Input shape (batchless): 3 × 224 × 224
- Output shape: 1000
- Verifier verdict: warn
- Graph JSON: https://neurarch.com/templates/swin-tiny/model.json
- Open on the canvas: https://neurarch.com/?template=swin-tiny

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | image | Input | shape=[3, 224, 224] | 3 × 224 × 224 |
| 2 | patch_embed_4x4 | Patch Embed | embedDim=96, patchSize=4 | 3136 × 96 |
| 3 | s1b1_norm1 | LayerNorm | normalizedShape=96 | 3136 × 96 |
| 4 | s1b1_window_attn | Multi-Head Attention | embedDim=96, numHeads=3 | 3136 × 96 |
| 5 | s1b1_res1 | Add |  | 3136 × 96 |
| 6 | s1b1_norm2 | LayerNorm | normalizedShape=96 | 3136 × 96 |
| 7 | s1b1_mlp | Feed Forward | embedDim=96, ffDim=384 | 3136 × 96 |
| 8 | s1b1_res2 | Add |  | 3136 × 96 |
| 9 | s1b2_norm1 | LayerNorm | normalizedShape=96 | 3136 × 96 |
| 10 | s1b2_shifted_window_attn | Multi-Head Attention | embedDim=96, numHeads=3 | 3136 × 96 |
| 11 | s1b2_res1 | Add |  | 3136 × 96 |
| 12 | s1b2_norm2 | LayerNorm | normalizedShape=96 | 3136 × 96 |
| 13 | s1b2_mlp | Feed Forward | embedDim=96, ffDim=384 | 3136 × 96 |
| 14 | s1b2_res2 | Add |  | 3136 × 96 |
| 15 | patch_merging_2 | Linear | outFeatures=192, inFeatures=384 | 3136 × 192 |
| 16 | s2b1_norm1 | LayerNorm | normalizedShape=192 | 3136 × 192 |
| 17 | s2b1_window_attn | Multi-Head Attention | embedDim=192, numHeads=6 | 3136 × 192 |
| 18 | s2b1_res1 | Add |  | 3136 × 192 |
| 19 | s2b1_norm2 | LayerNorm | normalizedShape=192 | 3136 × 192 |
| 20 | s2b1_mlp | Feed Forward | embedDim=192, ffDim=768 | 3136 × 192 |
| 21 | s2b1_res2 | Add |  | 3136 × 192 |
| 22 | s2b2_norm1 | LayerNorm | normalizedShape=192 | 3136 × 192 |
| 23 | s2b2_shifted_window_attn | Multi-Head Attention | embedDim=192, numHeads=6 | 3136 × 192 |
| 24 | s2b2_res1 | Add |  | 3136 × 192 |
| 25 | s2b2_norm2 | LayerNorm | normalizedShape=192 | 3136 × 192 |
| 26 | s2b2_mlp | Feed Forward | embedDim=192, ffDim=768 | 3136 × 192 |
| 27 | s2b2_res2 | Add |  | 3136 × 192 |
| 28 | patch_merging_3 | Linear | outFeatures=384, inFeatures=768 | 3136 × 384 |
| 29 | s3b1_norm1 | LayerNorm | normalizedShape=384 | 3136 × 384 |
| 30 | s3b1_window_attn | Multi-Head Attention | embedDim=384, numHeads=12 | 3136 × 384 |
| 31 | s3b1_res1 | Add |  | 3136 × 384 |
| 32 | s3b1_norm2 | LayerNorm | normalizedShape=384 | 3136 × 384 |
| 33 | s3b1_mlp | Feed Forward | embedDim=384, ffDim=1536 | 3136 × 384 |
| 34 | s3b1_res2 | Add |  | 3136 × 384 |
| 35 | s3b2_norm1 | LayerNorm | normalizedShape=384 | 3136 × 384 |
| 36 | s3b2_shifted_window_attn | Multi-Head Attention | embedDim=384, numHeads=12 | 3136 × 384 |
| 37 | s3b2_res1 | Add |  | 3136 × 384 |
| 38 | s3b2_norm2 | LayerNorm | normalizedShape=384 | 3136 × 384 |
| 39 | s3b2_mlp | Feed Forward | embedDim=384, ffDim=1536 | 3136 × 384 |
| 40 | s3b2_res2 | Add |  | 3136 × 384 |
| 41 | s3b3_norm1 | LayerNorm | normalizedShape=384 | 3136 × 384 |
| 42 | s3b3_window_attn | Multi-Head Attention | embedDim=384, numHeads=12 | 3136 × 384 |
| 43 | s3b3_res1 | Add |  | 3136 × 384 |
| 44 | s3b3_norm2 | LayerNorm | normalizedShape=384 | 3136 × 384 |
| 45 | s3b3_mlp | Feed Forward | embedDim=384, ffDim=1536 | 3136 × 384 |
| 46 | s3b3_res2 | Add |  | 3136 × 384 |
| 47 | s3b4_norm1 | LayerNorm | normalizedShape=384 | 3136 × 384 |
| 48 | s3b4_shifted_window_attn | Multi-Head Attention | embedDim=384, numHeads=12 | 3136 × 384 |
| 49 | s3b4_res1 | Add |  | 3136 × 384 |
| 50 | s3b4_norm2 | LayerNorm | normalizedShape=384 | 3136 × 384 |
| 51 | s3b4_mlp | Feed Forward | embedDim=384, ffDim=1536 | 3136 × 384 |
| 52 | s3b4_res2 | Add |  | 3136 × 384 |
| 53 | s3b5_norm1 | LayerNorm | normalizedShape=384 | 3136 × 384 |
| 54 | s3b5_window_attn | Multi-Head Attention | embedDim=384, numHeads=12 | 3136 × 384 |
| 55 | s3b5_res1 | Add |  | 3136 × 384 |
| 56 | s3b5_norm2 | LayerNorm | normalizedShape=384 | 3136 × 384 |
| 57 | s3b5_mlp | Feed Forward | embedDim=384, ffDim=1536 | 3136 × 384 |
| 58 | s3b5_res2 | Add |  | 3136 × 384 |
| 59 | s3b6_norm1 | LayerNorm | normalizedShape=384 | 3136 × 384 |
| 60 | s3b6_shifted_window_attn | Multi-Head Attention | embedDim=384, numHeads=12 | 3136 × 384 |
| 61 | s3b6_res1 | Add |  | 3136 × 384 |
| 62 | s3b6_norm2 | LayerNorm | normalizedShape=384 | 3136 × 384 |
| 63 | s3b6_mlp | Feed Forward | embedDim=384, ffDim=1536 | 3136 × 384 |
| 64 | s3b6_res2 | Add |  | 3136 × 384 |
| 65 | patch_merging_4 | Linear | outFeatures=768, inFeatures=1536 | 3136 × 768 |
| 66 | s4b1_norm1 | LayerNorm | normalizedShape=768 | 3136 × 768 |
| 67 | s4b1_window_attn | Multi-Head Attention | embedDim=768, numHeads=24 | 3136 × 768 |
| 68 | s4b1_res1 | Add |  | 3136 × 768 |
| 69 | s4b1_norm2 | LayerNorm | normalizedShape=768 | 3136 × 768 |
| 70 | s4b1_mlp | Feed Forward | embedDim=768, ffDim=3072 | 3136 × 768 |
| 71 | s4b1_res2 | Add |  | 3136 × 768 |
| 72 | s4b2_norm1 | LayerNorm | normalizedShape=768 | 3136 × 768 |
| 73 | s4b2_shifted_window_attn | Multi-Head Attention | embedDim=768, numHeads=24 | 3136 × 768 |
| 74 | s4b2_res1 | Add |  | 3136 × 768 |
| 75 | s4b2_norm2 | LayerNorm | normalizedShape=768 | 3136 × 768 |
| 76 | s4b2_mlp | Feed Forward | embedDim=768, ffDim=3072 | 3136 × 768 |
| 77 | s4b2_res2 | Add |  | 3136 × 768 |
| 78 | final_norm | LayerNorm | normalizedShape=768 | 3136 × 768 |
| 79 | avgpool | GlobalAvgPool1D |  | 3136 |
| 80 | classifier | Linear | outFeatures=1000, inFeatures=768 | 1000 |
| 81 | class_logits | Output |  | 1000 |

## Verifier findings

- **warn** `attention-no-pe` at `s1b1_window_attn`: 12 attention layer(s) present but no positional encoding found. Attention is permutation-invariant, without position information the model cannot distinguish token order. Fix: Add a PositionalEncoding (sinusoidal) or RoPE layer before the first attention layer.
- **info** `deep-attention-default-init` at `s1b1_window_attn`: At 12 stacked attention layers, residual-branch outputs add up; unscaled init lets activation variance grow with depth. GPT-2/LLaMA-family models scale the residual projections by depth (N(0, 0.02 / √(2L))). Fix: Scale residual output projections by depth: nn.init.normal_(w, std=0.02 / math.sqrt(2 * n_layers))

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)
#
# WARNING: 1 layer(s) below are not yet supported by the PyTorch
# exporter and pass their input through UNCHANGED in forward():
#   - avgpool (globalAvgPool1d)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class SwinTransformerTiny(nn.Module):
    def __init__(self):
        super().__init__()

        self.patchEmbed_1 = nn.Conv2d(3, 96, kernel_size=4, stride=4)  # Patch embedding (ViT-style)
        self.layerNorm_1 = nn.LayerNorm(96)
        self.multiHeadAttention_1 = nn.MultiheadAttention(embed_dim=96, num_heads=3, batch_first=True)
        self.layerNorm_2 = nn.LayerNorm(96)
        self.feedForward_1 = nn.Sequential(
            nn.Linear(96, 384),
            nn.ReLU(),
            nn.Linear(384, 96)
        )
        self.layerNorm_3 = nn.LayerNorm(96)
        self.multiHeadAttention_2 = nn.MultiheadAttention(embed_dim=96, num_heads=3, batch_first=True)
        self.layerNorm_4 = nn.LayerNorm(96)
        self.feedForward_2 = nn.Sequential(
            nn.Linear(96, 384),
            nn.ReLU(),
            nn.Linear(384, 96)
        )
        self.linear_1 = nn.Linear(384, 192)
        self.layerNorm_5 = nn.LayerNorm(192)
        self.multiHeadAttention_3 = nn.MultiheadAttention(embed_dim=192, num_heads=6, batch_first=True)
        self.layerNorm_6 = nn.LayerNorm(192)
        self.feedForward_3 = nn.Sequential(
            nn.Linear(192, 768),
            nn.ReLU(),
            nn.Linear(768, 192)
        )
        self.layerNorm_7 = nn.LayerNorm(192)
        self.multiHeadAttention_4 = nn.MultiheadAttention(embed_dim=192, num_heads=6, batch_first=True)
        self.layerNorm_8 = nn.LayerNorm(192)
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md