# Search-based Interest Model (SIM)

> Alibaba 2020 - a lifelong sequence (2000 events) is cut to the 50 most relevant before anything quadratic runs, then attended against the candidate.

Pick when the behaviour history is thousands of events long and full attention over it is unaffordable. The retrieval step is what makes the cost linear again.

- Category: Recommendation
- Paper: Pi et al. (2020). Search-based User Interest Modeling with Lifelong Sequential Behavior Data for Click-Through Rate Prediction. CIKM 2020 (https://arxiv.org/abs/2006.05639)
- Layers: 16
- Parameters: 128.06M
- Input shape (batchless): 2000
- Output shape: 1
- Verifier verdict: pass
- Graph JSON: https://neurarch.com/templates/sim/model.json
- Open on the canvas: https://neurarch.com/?template=sim

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | Lifelong Seq (2000 events) | Input | shape=[2000] | 2000 |
| 2 | Item Embed | Embedding | vocabSize=1000000 | 2000 × 64 |
| 3 | Candidate Item | Input | shape=[1] | 1 |
| 4 | Candidate Embed | Embedding | vocabSize=1000000 | 1 × 64 |
| 5 | GSU: top-50 by relevance | Long-Sequence Retrieval (SIM/ETA GSU) | embedDim=64, topK=50 | 50 × 64 |
| 6 | ESU: multi-head target attention | Target Attention (DIN) | embedDim=64 | 64 |
| 7 | User Profile | Input | shape=[16] | 16 |
| 8 | User Tower | Linear | outFeatures=32, inFeatures=16 | 32 |
| 9 | [interest; user] | Concatenate |  | 96 |
| 10 | MLP 200 | Linear | outFeatures=200, inFeatures=96 | 200 |
| 11 | Dice (~PReLU) | PReLU |  | 200 |
| 12 | MLP 80 | Linear | outFeatures=80, inFeatures=200 | 80 |
| 13 | Dice (~PReLU) | PReLU |  | 80 |
| 14 | CTR Head | Linear | outFeatures=1, inFeatures=80 | 1 |
| 15 | Sigmoid | Sigmoid |  | 1 |
| 16 | pCTR | Output |  | 1 |

## Verifier findings

- **info** `output-activation` at `Sigmoid`: "Sigmoid" feeds directly into Output. PyTorch's nn.CrossEntropyLoss already applies log-softmax internally, an explicit Softmax causes double-application and degrades training stability. Fix: Remove Softmax/Sigmoid for training. Restore it in a separate inference wrapper or ONNX export.
- **info** `vanishing-gradient` at `Sigmoid`: Sigmoid saturates to [0,1] / [-1,1], and its gradient approaches zero for large inputs. In networks deeper than 5 layers, this halts learning in early layers. Fix: Use ReLU, GELU, or SiLU for hidden layers. Keep Sigmoid only at binary classification outputs; Tanh in specific contexts (GAN generators, LSTM gates).
- **info** `deep-no-norm`: 12 layers with no BatchNorm, LayerNorm, or GroupNorm. Without normalization, activations can explode or vanish across layers, causing slow or unstable training. Fix: Add BatchNorm after Conv2d (CV tasks), LayerNorm after attention/FFN (NLP/LLM), or GroupNorm for small batch sizes.
- **info** `init-activation-mismatch` at `CTR Head`: PyTorch initializes Linear/Conv with Kaiming (He) init, which is derived for ReLU-family activations. Feeding a saturating activation (sigmoid/tanh) from a He-initialized layer starts training in the saturated tails, shrinking early gradients. Fix: Initialize these layers with Xavier instead: nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain("sigmoid"|"tanh")), or switch the activation to a ReLU-family one.

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class TargetAttention(nn.Module):
    """DIN's local activation unit. The CANDIDATE is the query over the
    behaviour sequence, so the sequence axis is consumed and one interest
    vector comes out per candidate."""

    def __init__(self, embed_dim: int, hidden_dim: int = 36):
        super().__init__()
        self.mlp = nn.Sequential(
            nn.Linear(4 * embed_dim, hidden_dim),
            nn.PReLU(),
            nn.Linear(hidden_dim, 1),
        )

    def forward(self, query: torch.Tensor, keys: torch.Tensor) -> torch.Tensor:
        if query.dim() == keys.dim() - 1:
            query = query.unsqueeze(-2)
        q = query[..., :1, :].expand_as(keys)
        feats = torch.cat([q, keys, q - keys, q * keys], dim=-1)
        w = self.mlp(feats).softmax(dim=-2)
        return (w * keys).sum(dim=-2)


class BehaviorRetrieval(nn.Module):
    """SIM / ETA general search unit: cut a lifelong behaviour sequence to its
    top-k most relevant events BEFORE anything quadratic runs. Hard mode is a
    category lookup and learns nothing."""

    def __init__(self, top_k: int = 50, embed_dim: int = 64, mode: str = "soft"):
        super().__init__()
        self.top_k = top_k
        self.mode = mode
        self.proj = None if mode == "hard" else nn.Linear(embed_dim, embed_dim, bias=False)

    def forward(self, seq: torch.Tensor, target=None) -> torch.Tensor:
        k = min(self.top_k, seq.size(-2))
        h = seq if self.proj is None else self.proj(seq)
        if target is None:
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md