# Wide & Deep

> Memorization (wide linear) + generalization (deep MLP) joint trained — Cheng et al. 2016

Pick as a simpler, more interpretable alternative to DLRM for CTR. Linear part captures memorized rules, deep part generalizes.

- Category: Recommendation
- Layers: 13
- Parameters: 3.65M
- Input shape (batchless): 10000
- Output shape: 1
- Verifier verdict: pass
- Graph JSON: https://neurarch.com/templates/wide-and-deep/model.json
- Open on the canvas: https://neurarch.com/?template=wide-and-deep

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | Wide Input (cross feats) | Input | shape=[10000] | 10000 |
| 2 | Wide Linear | Linear | outFeatures=1, inFeatures=10000 | 1 |
| 3 | Deep Input (sparse cat) | Input | shape=[50] | 50 |
| 4 | Embeddings | Embedding | vocabSize=100000 | 50 × 32 |
| 5 | Flatten | Flatten |  | 1600 |
| 6 | Deep FC 1 | Linear | outFeatures=256, inFeatures=1600 | 256 |
| 7 | ReLU 1 | ReLU |  | 256 |
| 8 | Deep FC 2 | Linear | outFeatures=128, inFeatures=256 | 128 |
| 9 | ReLU 2 | ReLU |  | 128 |
| 10 | Deep Out | Linear | outFeatures=1, inFeatures=128 | 1 |
| 11 | Wide + Deep | Add |  | 1 |
| 12 | Sigmoid CTR | Sigmoid |  | 1 |
| 13 | P(click) | Output |  | 1 |

## Verifier findings

- **info** `output-activation` at `Sigmoid CTR`: "Sigmoid CTR" feeds directly into Output. PyTorch's nn.CrossEntropyLoss already applies log-softmax internally, an explicit Softmax causes double-application and degrades training stability. Fix: Remove Softmax/Sigmoid for training. Restore it in a separate inference wrapper or ONNX export.
- **info** `vanishing-gradient` at `Sigmoid CTR`: Sigmoid saturates to [0,1] / [-1,1], and its gradient approaches zero for large inputs. In networks deeper than 5 layers, this halts learning in early layers. Fix: Use ReLU, GELU, or SiLU for hidden layers. Keep Sigmoid only at binary classification outputs; Tanh in specific contexts (GAN generators, LSTM gates).
- **info** `deep-no-norm`: 10 layers with no BatchNorm, LayerNorm, or GroupNorm. Without normalization, activations can explode or vanish across layers, causing slow or unstable training. Fix: Add BatchNorm after Conv2d (CV tasks), LayerNorm after attention/FFN (NLP/LLM), or GroupNorm for small batch sizes.

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class WideDeep(nn.Module):
    def __init__(self):
        super().__init__()

        self.linear_1 = nn.Linear(10000, 1)
        self.embedding_1 = nn.Embedding(100000, 32)
        self.linear_2 = nn.Linear(1600, 256)
        self.linear_3 = nn.Linear(256, 128)
        self.linear_4 = nn.Linear(128, 1)

    def forward(self, src, tgt=None):
        # Wide Input (cross feats) shape: [10000]
        # Deep Input (sparse cat) shape: [50]
        linear_ide_fc = self.linear_1(src)
        embedding_ep_emb = self.embedding_1(tgt)
        flatten_p_flat = torch.flatten(embedding_ep_emb, 1)
        linear_ep_fc1 = self.linear_2(flatten_p_flat)
        relu__relu1 = F.relu(linear_ep_fc1)
        linear_ep_fc2 = self.linear_3(relu__relu1)
        relu__relu2 = F.relu(linear_ep_fc2)
        linear_ep_out = self.linear_4(relu__relu2)
        add_merge = linear_ide_fc + linear_ep_out
        sigmoid_igmoid = torch.sigmoid(add_merge)
        # Output
        return sigmoid_igmoid


if __name__ == '__main__':
    model = WideDeep()
    model.eval()

    src = torch.randint(0, 10000, (1, 10000))  # (batch, src_seq_len)
    tgt = torch.randint(0, 50, (1, 50))  # (batch, tgt_seq_len)
    with torch.no_grad():
        output = model(src, tgt)

    print(f'Src shape    : {tuple(src.shape)}')
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md