# DLRM vs Wide & Deep

Two production ranking models with different interaction layers.

**Wide & Deep has 61M fewer parameters than DLRM: 3 layers added, 4 removed, 6 changed.**

Source: https://neurarch.com/diff/dlrm-vs-wide-and-deep.html

## Sides

| | DLRM | Wide & Deep |
|---|---|---|
| Layers | 11 | 10 |
| Parameters | 64M | 3.7M |
| Input | 13 | 10000 |
| Output | 1 | 1 |
| Forward-passes | yes | yes |
| Est. train cost | $0.063 | $0.043 |
| T4 16GB | fits | fits |
| A100 40GB | fits | fits |
| H100 80GB | fits | fits |

## Deltas (Wide & Deep relative to DLRM)

- Parameters: -61M (-94.3%)
- Layers: -1
- Added 3, removed 4, changed 6, unchanged 4

## Layer by layer

| # | Status | DLRM | Params | Output | Wide & Deep | Params | Output |
|---|---|---|---|---|---|---|---|
| 1 | changed (shape) | Dense Features (Input) |  | 13 | Wide Input (cross feats) (Input) |  | 10000 |
| 2 | changed (shape) | Sparse Features (Input) |  | 26 | Deep Input (sparse cat) (Input) |  | 50 |
| 3 | removed | Bottom MLP 1 (Linear) | 896 | 64 |  | |  |
| 4 | removed | EmbeddingBag (Embedding Bag) | 64M | 26 × 64 |  | |  |
| 5 | removed | ReLU (Relu) |  | 64 |  | |  |
| 6 | changed (inFeatures, outFeatures) | Bottom MLP 2 (Linear) | 4.2K | 64 | Wide Linear (Linear) | 10K | 1 |
| 7 | removed | Feature Interaction (Feature Interaction) |  | 415 |  | |  |
| 8 | added |  | |  | Embeddings (Embedding) | 3.2M | 50 × 32 |
| 9 | added |  | |  | Flatten (Flatten) |  | 1600 |
| 10 | changed (inFeatures, outFeatures) | Top MLP 1 (Linear) | 213K | 512 | Deep FC 1 (Linear) | 410K | 256 |
| 11 | same | ReLU (Relu) |  | 512 | ReLU 1 (Relu) |  | 256 |
| 12 | changed (inFeatures, outFeatures) | Top MLP 2 (Linear) | 131K | 256 | Deep FC 2 (Linear) | 33K | 128 |
| 13 | same | ReLU (Relu) |  | 256 | ReLU 2 (Relu) |  | 128 |
| 14 | changed (inFeatures) | CTR Head (Linear) | 257 | 1 | Deep Out (Linear) | 129 | 1 |
| 15 | added |  | |  | Wide + Deep (Add) |  | 1 |
| 16 | same | Sigmoid (Sigmoid) |  | 1 | Sigmoid CTR (Sigmoid) |  | 1 |
| 17 | same | P(click) (Output) |  | 1 | P(click) (Output) |  | 1 |

## What this is not

- The two are priced at different declared inputs (13 against 10000), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

## Graphs

- DLRM: https://neurarch.com/templates/dlrm/model.json
- Wide & Deep: https://neurarch.com/templates/wide-and-deep/model.json
- Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`