# ResNet Block vs Simple CNN

What the residual connection costs and what it buys.

**Simple CNN has 731K more parameters than ResNet Block: 4 layers added, 4 removed, 2 changed.**

Source: https://neurarch.com/diff/resnet-block-vs-simple-cnn.html

## Sides

| | ResNet Block | Simple CNN |
|---|---|---|
| Layers | 7 | 7 |
| Parameters | 74K | 805K |
| Input | 64 × 32 × 32 | 1 × 28 × 28 |
| Output | 64 × 32 × 32 | 10 |
| Forward-passes | yes | yes |
| Est. train cost | $0.045 | $0.042 |
| T4 16GB | fits | fits |
| A100 40GB | fits | fits |
| H100 80GB | fits | fits |

## Deltas (Simple CNN relative to ResNet Block)

- Parameters: +731K (11× the size)
- Layers: 0
- Added 4, removed 4, changed 2, unchanged 3

## Layer by layer

| # | Status | ResNet Block | Params | Output | Simple CNN | Params | Output |
|---|---|---|---|---|---|---|---|
| 1 | changed (shape) | Input (Input) |  | 64 × 32 × 32 | Input (Input) |  | 1 × 28 × 28 |
| 2 | changed (outChannels) | Conv2D_1 (Conv2d) | 640 | 64 × 32 × 32 | Conv2D_1 (Conv2d) | 320 | 32 × 28 × 28 |
| 3 | removed | BatchNorm_1 (Batch Norm) |  | 64 × 32 × 32 |  | |  |
| 4 | same | ReLU_1 (Relu) |  | 64 × 32 × 32 | ReLU_1 (Relu) |  | 32 × 28 × 28 |
| 5 | removed | Conv2D_2 (Conv2d) | 640 | 64 × 32 × 32 |  | |  |
| 6 | removed | BatchNorm_2 (Batch Norm) |  | 64 × 32 × 32 |  | |  |
| 7 | removed | Add (Add) |  | 64 × 32 × 32 |  | |  |
| 8 | added |  | |  | MaxPool2D_1 (Maxpool2d) |  | 32 × 14 × 14 |
| 9 | added |  | |  | Flatten (Flatten) |  | 6272 |
| 10 | added |  | |  | Linear_1 (Linear) |  | 128 |
| 11 | same | ReLU_2 (Relu) |  | 64 × 32 × 32 | ReLU_2 (Relu) |  | 128 |
| 12 | added |  | |  | Linear_2 (Linear) |  | 10 |
| 13 | same | Output (Output) |  | 64 × 32 × 32 | Output (Output) |  | 10 |

## What this is not

- The two are priced at different declared inputs (64 × 32 × 32 against 1 × 28 × 28), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

## Graphs

- ResNet Block: https://neurarch.com/templates/resnet-block/model.json
- Simple CNN: https://neurarch.com/templates/simple-cnn/model.json
- Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`