# Multi-Interest Network (MIND)

> Alibaba 2019 - dynamic routing turns one behaviour sequence into K interest vectors, so a user is K points in item space rather than one.

Pick for candidate retrieval when users have several distinct intents and a single user vector averages them into nothing. Note every downstream layer gains a rank.

- Category: Recommendation
- Paper: Li et al. (2019). Multi-Interest Network with Dynamic Routing for Recommendation at Tmall. CIKM 2019 (https://arxiv.org/abs/1904.08030)
- Layers: 12
- Parameters: 128.01M
- Input shape (batchless): 50
- Output shape: 1
- Verifier verdict: pass
- Graph JSON: https://neurarch.com/templates/mind/model.json
- Open on the canvas: https://neurarch.com/?template=mind

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | Behavior Seq (50 items) | Input | shape=[50] | 50 |
| 2 | Item Embed | Embedding | vocabSize=1000000 | 50 × 64 |
| 3 | Dynamic Routing (K=4) | Multi-Interest Extractor (MIND/ComiRec) | embedDim=64 | 4 × 64 |
| 4 | Interest Projection | Linear | outFeatures=64, inFeatures=64 | 4 × 64 |
| 5 | Candidate Item | Input | shape=[1] | 1 |
| 6 | Item Embed (shared vocab) | Embedding | vocabSize=1000000 | 1 × 64 |
| 7 | transpose | Permute |  | 64 × 1 |
| 8 | interest · item | MatMul |  | 4 × 1 |
| 9 | Label-aware max over K | TopK |  | 1 × 1 |
| 10 | flatten | Flatten |  | 1 |
| 11 | Sigmoid | Sigmoid |  | 1 |
| 12 | score | Output |  | 1 |

## Verifier findings

- **info** `output-activation` at `Sigmoid`: "Sigmoid" feeds directly into Output. PyTorch's nn.CrossEntropyLoss already applies log-softmax internally, an explicit Softmax causes double-application and degrades training stability. Fix: Remove Softmax/Sigmoid for training. Restore it in a separate inference wrapper or ONNX export.
- **info** `vanishing-gradient` at `Sigmoid`: Sigmoid saturates to [0,1] / [-1,1], and its gradient approaches zero for large inputs. In networks deeper than 5 layers, this halts learning in early layers. Fix: Use ReLU, GELU, or SiLU for hidden layers. Keep Sigmoid only at binary classification outputs; Tanh in specific contexts (GAN generators, LSTM gates).
- **info** `deep-no-norm`: 9 layers with no BatchNorm, LayerNorm, or GroupNorm. Without normalization, activations can explode or vanish across layers, causing slow or unstable training. Fix: Add BatchNorm after Conv2d (CV tasks), LayerNorm after attention/FFN (NLP/LLM), or GroupNorm for small batch sizes.

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class MultiInterest(nn.Module):
    """MIND behaviour-to-interest dynamic routing. One user becomes K vectors,
    not one, so every downstream layer sees a rank it has to agree with."""

    def __init__(self, embed_dim: int, num_interests: int = 4, num_iterations: int = 3):
        super().__init__()
        self.k = num_interests
        self.iters = max(1, num_iterations)
        self.bilinear = nn.Linear(embed_dim, embed_dim, bias=False)

    @staticmethod
    def _squash(x: torch.Tensor) -> torch.Tensor:
        n2 = x.pow(2).sum(-1, keepdim=True)
        return (n2 / (1.0 + n2)) * x / (n2.sqrt() + 1e-8)

    def forward(self, seq: torch.Tensor) -> torch.Tensor:
        u = self.bilinear(seq)
        b = seq.new_zeros(seq.size(0), self.k, seq.size(-2))
        v = self._squash(torch.bmm(b.softmax(dim=1), u))
        for _ in range(self.iters - 1):
            b = b + torch.bmm(v, u.transpose(1, 2))
            v = self._squash(torch.bmm(b.softmax(dim=1), u))
        return v


class Multi_InterestNetworkMIND(nn.Module):
    def __init__(self):
        super().__init__()

        self.embedding_1 = nn.Embedding(1000000, 64)
        self.multiInterest_1 = MultiInterest(embed_dim=64, num_interests=4, num_iterations=3)
        self.linear_1 = nn.Linear(64, 64)
        self.embedding_2 = nn.Embedding(1000000, 64)

    def forward(self, src, tgt=None):
        # Behavior Seq (50 items) shape: [50]
        # Candidate Item shape: [1]
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md