# SLi-Rec

> Yu et al. 2019 — Short and Long-term Interest Recommender. Time-LSTM + ASVD attention fused via gate

Pick when both short-session intent and long-term preferences matter and you want each modelled separately, then fused via a learned gate.

- Category: Recommendation
- Layers: 20
- Parameters: 64.05M
- Input shape (batchless): 50
- Output shape: 1
- Verifier verdict: warn
- Graph JSON: https://neurarch.com/templates/sli-rec/model.json
- Open on the canvas: https://neurarch.com/?template=sli-rec

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | User History (T items) | Input | shape=[50] | 50 |
| 2 | Item Embed | Embedding | vocabSize=1000000 | 50 × 32 |
| 3 | Time-LSTM (short) | LSTM | inFeatures=32, hiddenSize=64, numLayers=1 | 64 |
| 4 | ASVD Attn (long) | Self-Attention | numHeads=4 | 50 × 32 |
| 5 | to_channels | Permute |  | 32 × 50 |
| 6 | Pool Long | GlobalAvgPool1D |  | 32 |
| 7 | Long Proj | Linear | outFeatures=64, inFeatures=32 | 64 |
| 8 | Target Item | Input | shape=[1] | 1 |
| 9 | Target Embed | Embedding | vocabSize=1000000 | 1 × 32 |
| 10 | Flatten | Flatten |  | 32 |
| 11 | Target Proj | Linear | outFeatures=64, inFeatures=32 | 64 |
| 12 | Fusion Gate σ(W·[s;l;t]) | Linear | outFeatures=64, inFeatures=192 | 64 |
| 13 | Gate σ | Sigmoid |  | 64 |
| 14 | α·short + (1−α)·long | Add |  | 64 |
| 15 | Concat [user, target] | Concatenate |  | 128 |
| 16 | MLP 1 | Linear | outFeatures=64, inFeatures=128 | 64 |
| 17 | PReLU | PReLU |  | 64 |
| 18 | Score Head | Linear | outFeatures=1, inFeatures=64 | 1 |
| 19 | Sigmoid | Sigmoid |  | 1 |
| 20 | P(click) | Output |  | 1 |

## Verifier findings

- **info** `output-activation` at `Sigmoid`: "Sigmoid" feeds directly into Output. PyTorch's nn.CrossEntropyLoss already applies log-softmax internally, an explicit Softmax causes double-application and degrades training stability. Fix: Remove Softmax/Sigmoid for training. Restore it in a separate inference wrapper or ONNX export.
- **warn** `attention-no-pe` at `ASVD Attn (long)`: 1 attention layer(s) present but no positional encoding found. Attention is permutation-invariant, without position information the model cannot distinguish token order. Fix: Add a PositionalEncoding (sinusoidal) or RoPE layer before the first attention layer.
- **info** `vanishing-gradient` at `Gate σ`: Sigmoid saturates to [0,1] / [-1,1], and its gradient approaches zero for large inputs. In networks deeper than 5 layers, this halts learning in early layers. Fix: Use ReLU, GELU, or SiLU for hidden layers. Keep Sigmoid only at binary classification outputs; Tanh in specific contexts (GAN generators, LSTM gates).
- **info** `vanishing-gradient` at `Sigmoid`: Sigmoid saturates to [0,1] / [-1,1], and its gradient approaches zero for large inputs. In networks deeper than 5 layers, this halts learning in early layers. Fix: Use ReLU, GELU, or SiLU for hidden layers. Keep Sigmoid only at binary classification outputs; Tanh in specific contexts (GAN generators, LSTM gates).
- **info** `deep-no-norm`: 17 layers with no BatchNorm, LayerNorm, or GroupNorm. Without normalization, activations can explode or vanish across layers, causing slow or unstable training. Fix: Add BatchNorm after Conv2d (CV tasks), LayerNorm after attention/FFN (NLP/LLM), or GroupNorm for small batch sizes.
- **info** `consecutive-linear-no-activation` at `Long Proj`: "Long Proj" feeds directly into "Fusion Gate σ(W·[s;l;t])" with no activation between them. Two stacked linear maps collapse into one (W₂·W₁), so the extra layer costs parameters but adds no representational power. Fix: Add a non-linearity (ReLU/GELU) between them. If this is a deliberate low-rank / factorized projection (down-proj → up-proj), this hint is safe to ignore.
- **info** `consecutive-linear-no-activation` at `Target Proj`: "Target Proj" feeds directly into "Fusion Gate σ(W·[s;l;t])" with no activation between them. Two stacked linear maps collapse into one (W₂·W₁), so the extra layer costs parameters but adds no representational power. Fix: Add a non-linearity (ReLU/GELU) between them. If this is a deliberate low-rank / factorized projection (down-proj → up-proj), this hint is safe to ignore.
- **info** `init-activation-mismatch` at `Fusion Gate σ(W·[s;l;t])`: PyTorch initializes Linear/Conv with Kaiming (He) init, which is derived for ReLU-family activations. Feeding a saturating activation (sigmoid/tanh) from a He-initialized layer starts training in the saturated tails, shrinking early gradients. Fix: Initialize these layers with Xavier instead: nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain("sigmoid"|"tanh")), or switch the activation to a ReLU-family one.

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)
#
# WARNING: 2 layer(s) below are not yet supported by the PyTorch
# exporter and pass their input through UNCHANGED in forward():
#   - to_channels (permute)
#   - Pool Long (globalAvgPool1d)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class SLi_Rec(nn.Module):
    def __init__(self):
        super().__init__()

        self.embedding_1 = nn.Embedding(1000000, 32)
        self.lstm_1 = nn.LSTM(32, 64, num_layers=1, batch_first=True)
        self.selfAttention_1 = nn.MultiheadAttention(embed_dim=128, num_heads=4, batch_first=True)
        self.linear_1 = nn.Linear(32, 64)
        self.embedding_2 = nn.Embedding(1000000, 32)
        self.linear_2 = nn.Linear(32, 64)
        self.linear_3 = nn.Linear(192, 64)
        self.linear_4 = nn.Linear(128, 64)
        self.prelu_1 = nn.PReLU(num_parameters=1)
        self.linear_5 = nn.Linear(64, 1)

    def forward(self, src, tgt=None):
        # User History (T items) shape: [50]
        # Target Item shape: [1]
        embedding_st_emb = self.embedding_1(src)
        lstm_lstm = self.lstm_1(embedding_st_emb)[0][:, -1, :]
        self_attention_g_attn = self.selfAttention_1(embedding_st_emb, embedding_st_emb, embedding_st_emb)[0]
        # TODO: layer 'to_channels' (permute) is not yet supported by the exporter; passing through unchanged
        # TODO: layer 'Pool Long' (globalAvgPool1d) is not yet supported by the exporter; passing through unchanged
        linear_ong_fc = self.linear_1(self_attention_g_attn)
        embedding_gt_emb = self.embedding_2(tgt)
        flatten_t_flat = torch.flatten(embedding_gt_emb, 1)
        linear_tgt_fc = self.linear_2(flatten_t_flat)
        linear_gate = self.linear_3(lstm_lstm)
        sigmoid_te_sig = torch.sigmoid(linear_gate)
        add_fused = sigmoid_te_sig + lstm_lstm + linear_ong_fc
        concatenate__final = torch.cat([add_fused, linear_tgt_fc], dim=-1)
        linear_fc1 = self.linear_4(concatenate__final)
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md