# Behavior Sequence Transformer vs SLi-Rec

Two sequential recommenders with different notions of time.

**SLi-Rec has 65M fewer parameters than Behavior Sequence Transformer: 6 layers added, 5 removed, 10 changed.**

Source: https://neurarch.com/diff/bst-vs-sli-rec.html

## Sides

| | Behavior Sequence Transformer | SLi-Rec |
|---|---|---|
| Layers | 16 | 18 |
| Parameters | 129M | 64M |
| Input | 50 | 50 |
| Output | 1 | 1 |
| Forward-passes | yes | yes |
| Est. train cost | $0.083 | $0.063 |
| T4 16GB | fits | fits |
| A100 40GB | fits | fits |
| H100 80GB | fits | fits |

## Deltas (SLi-Rec relative to Behavior Sequence Transformer)

- Parameters: -65M (-50.3%)
- Layers: +2
- Added 6, removed 5, changed 10, unchanged 5

## Layer by layer

| # | Status | Behavior Sequence Transformer | Params | Output | SLi-Rec | Params | Output |
|---|---|---|---|---|---|---|---|
| 1 | removed | Behavior Seq (T items) (Input) |  | 50 |  | |  |
| 2 | changed (shape) | Target Item (Input) |  | 1 | User History (T items) (Input) |  | 50 |
| 3 | changed (shape) | User Profile (Input) |  | 16 | Target Item (Input) |  | 1 |
| 4 | changed (embeddingDim) | Item Embed (Embedding) | 64M | 50 × 64 | Item Embed (Embedding) | 32M | 50 × 32 |
| 5 | changed (embeddingDim) | Target Embed (Embedding) | 64M | 1 × 64 | Target Embed (Embedding) | 32M | 1 × 32 |
| 6 | removed | User Tower (Linear) | 544 | 32 |  | |  |
| 7 | removed | Pos Encoding (Positional Encoding) |  | 50 × 64 |  | |  |
| 8 | removed | [seq; tgt] (Concatenate) |  | 51 × 64 |  | |  |
| 9 | added |  | |  | Time-LSTM (short) (Lstm) | 17K | 64 |
| 10 | changed (type, dModel, numHeads, dFf, numLayers, causal) | Transformer Encoder (Transformer Block) |  | 51 × 64 | ASVD Attn (long) (Self Attention) |  | 50 × 32 |
| 11 | added |  | |  | Flatten (Flatten) |  | 32 |
| 12 | same | to_channels (Permute) |  | 64 × 51 | to_channels (Permute) |  | 32 × 50 |
| 13 | added |  | |  | Target Proj (Linear) | 2.1K | 64 |
| 14 | same | Mean Pool (Global Avg Pool1d) |  | 64 | Pool Long (Global Avg Pool1d) |  | 32 |
| 15 | added |  | |  | Long Proj (Linear) | 2.1K | 64 |
| 16 | changed (numInputs) | Concat all (Concatenate) |  | 96 | Concat [s; l; t] (Concatenate) |  | 192 |
| 17 | changed (inFeatures, outFeatures) | MLP 1024 (Linear) | 99K | 1024 | Fusion Gate σ(W·[s;l;t]) (Linear) | 12K | 64 |
| 18 | changed (type) | PReLU (Prelu) | 1 | 1024 | Gate σ (Sigmoid) |  | 64 |
| 19 | added |  | |  | α·short + (1−α)·long (Add) |  | 64 |
| 20 | added |  | |  | Concat [user, target] (Concatenate) |  | 128 |
| 21 | changed (inFeatures, outFeatures) | MLP 512 (Linear) | 525K | 512 | MLP 1 (Linear) | 8.3K | 64 |
| 22 | same | PReLU (Prelu) | 1 | 512 | PReLU (Prelu) | 1 | 64 |
| 23 | removed | MLP 256 (Linear) | 131K | 256 |  | |  |
| 24 | changed (inFeatures) | CTR Head (Linear) | 257 | 1 | Score Head (Linear) | 65 | 1 |
| 25 | same | Sigmoid (Sigmoid) |  | 1 | Sigmoid (Sigmoid) |  | 1 |
| 26 | same | P(click) (Output) |  | 1 | P(click) (Output) |  | 1 |

## What this is not

- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

## Graphs

- Behavior Sequence Transformer: https://neurarch.com/templates/bst/model.json
- SLi-Rec: https://neurarch.com/templates/sli-rec/model.json
- Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`