# 1D CNN + LSTM vs PatchTST

Convolution-plus-recurrence against patched attention for time series.

**PatchTST has 84K more parameters than 1D CNN + LSTM: 14 layers added, 9 removed, 4 changed.**

Source: https://neurarch.com/diff/cnn-lstm-1d-vs-patch-tst.html

## Sides

| | 1D CNN + LSTM | PatchTST |
|---|---|---|
| Layers | 12 | 17 |
| Parameters | 312K | 395K |
| Input | 12 × 5000 | 1 × 22 × 1000 |
| Output | 5 | 4 |
| Forward-passes | yes | yes |
| Est. train cost | $0.059 | $0.043 |
| T4 16GB | fits | fits |
| A100 40GB | fits | fits |
| H100 80GB | fits | fits |

## Deltas (PatchTST relative to 1D CNN + LSTM)

- Parameters: +84K (+26.8%)
- Layers: +5
- Added 14, removed 9, changed 4, unchanged 1

## Layer by layer

| # | Status | 1D CNN + LSTM | Params | Output | PatchTST | Params | Output |
|---|---|---|---|---|---|---|---|
| 1 | changed (shape) | ts_window (Input) |  | 12 × 5000 | ts_window (Input) |  | 1 × 22 × 1000 |
| 2 | removed | conv1 (Conv1d) | 512 | 64 × 5000 |  | |  |
| 3 | added |  | |  | patch_embed (Patch Embed) | 98K | 62 × 128 |
| 4 | added |  | |  | pos_embed (Positional Encoding) |  | 62 × 128 |
| 5 | changed (type, normalizedShape) | bn (Batch Norm) | 128 | 64 × 5000 | norm (Layer Norm) | 256 | 62 × 128 |
| 6 | removed | act (Relu) |  | 64 × 5000 |  | |  |
| 7 | removed | pool (Maxpool1d) |  | 64 × 2500 |  | |  |
| 8 | removed | conv2 (Conv1d) | 768 | 128 × 2500 |  | |  |
| 9 | added |  | |  | self_attn (Multi Head Attention) | 66K | 62 × 128 |
| 10 | added |  | |  | residual (Add) |  | 62 × 128 |
| 11 | changed (type) | bn (Batch Norm) | 256 | 128 × 2500 | norm (Layer Norm) | 256 | 62 × 128 |
| 12 | removed | act (Relu) |  | 128 × 2500 |  | |  |
| 13 | removed | pool (Maxpool1d) |  | 128 × 1250 |  | |  |
| 14 | removed | to_timesteps (Permute) |  | 1250 × 128 |  | |  |
| 15 | removed | lstm (Lstm) | 199K | 128 |  | |  |
| 16 | removed | drop (Dropout) |  | 128 |  | |  |
| 17 | added |  | |  | dense (Feed Forward) | 66K | 62 × 128 |
| 18 | added |  | |  | residual (Add) |  | 62 × 128 |
| 19 | added |  | |  | norm (Layer Norm) | 256 | 62 × 128 |
| 20 | added |  | |  | self_attn (Multi Head Attention) | 66K | 62 × 128 |
| 21 | added |  | |  | residual (Add) |  | 62 × 128 |
| 22 | added |  | |  | norm (Layer Norm) | 256 | 62 × 128 |
| 23 | added |  | |  | dense (Feed Forward) | 66K | 62 × 128 |
| 24 | added |  | |  | residual (Add) |  | 62 × 128 |
| 25 | added |  | |  | norm (Layer Norm) | 256 | 62 × 128 |
| 26 | added |  | |  | flatten (Flatten) |  | 7936 |
| 27 | changed (outFeatures) | classifier (Linear) |  | 5 | classifier (Linear) |  | 4 |
| 28 | same | logits (Output) |  | 5 | logits (Output) |  | 4 |

## What this is not

- The two are priced at different declared inputs (12 × 5000 against 1 × 22 × 1000), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

## Graphs

- 1D CNN + LSTM: https://neurarch.com/templates/cnn-lstm-1d/model.json
- PatchTST: https://neurarch.com/templates/patch-tst/model.json
- Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`