# HSTU (Generative Recommender)

> Meta 2024 - recommendation as sequential transduction over one interleaved item+action stream, with pointwise aggregated attention instead of softmax.

Pick when you want the scaling behaviour of a language model on recommendation data. Not a transformer block: the arithmetic differs and so does the cost.

- Category: Recommendation
- Paper: Zhai et al. (2024). Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations. ICML 2024 (https://arxiv.org/abs/2402.17152)
- Layers: 10
- Parameters: 210.05M
- Input shape (batchless): 1 × 1024
- Output shape: 1 × 1024 × 200000
- Verifier verdict: pass
- Graph JSON: https://neurarch.com/templates/hstu/model.json
- Open on the canvas: https://neurarch.com/?template=hstu

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | Interleaved item+action tokens | Input | shape=[1, 1024] | 1 × 1024 |
| 2 | Unified Token Embed | Embedding | vocabSize=200000 | 1 × 1024 × 512 |
| 3 | RoPE | RoPE |  | 1 × 1024 × 512 |
| 4 | HSTU Block 1 | HSTU Block | embedDim=512, numHeads=4 | 1 × 1024 × 512 |
| 5 | HSTU Block 2 | HSTU Block | embedDim=512, numHeads=4 | 1 × 1024 × 512 |
| 6 | HSTU Block 3 | HSTU Block | embedDim=512, numHeads=4 | 1 × 1024 × 512 |
| 7 | HSTU Block 4 | HSTU Block | embedDim=512, numHeads=4 | 1 × 1024 × 512 |
| 8 | RMSNorm | RMSNorm | normalizedShape=512 | 1 × 1024 × 512 |
| 9 | Retrieval Head (catalogue logits) | LM Head | hiddenSize=512, vocabSize=200000 | 1 × 1024 × 200000 |
| 10 | next-item logits | Output |  | 1 × 1024 × 200000 |

## Verifier findings

No finding. Shapes propagate end to end and no advisory rule fires.

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class HSTUBlock(nn.Module):
    """Hierarchical Sequential Transduction Unit (Zhai et al., ICML 2024).
    Pointwise aggregated attention: NO softmax over the sequence, and four
    projections (u, v, q, k) rather than a transformer block's attention plus
    a separate FFN. It is not a transformer block and does not cost like one."""

    def __init__(self, embed_dim: int, num_heads: int = 4,
                 linear_dim: int = 128, attn_dim: int = 128):
        super().__init__()
        self.h, self.dv, self.dqk = num_heads, linear_dim, attn_dim
        self.uvqk = nn.Linear(embed_dim, num_heads * (2 * linear_dim + 2 * attn_dim))
        self.out = nn.Linear(num_heads * linear_dim, embed_dim)
        self.norm = nn.LayerNorm(embed_dim)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        b, t, _ = x.shape
        h, dv, dqk = self.h, self.dv, self.dqk
        u, v, q, k = self.uvqk(x).split([h * dv, h * dv, h * dqk, h * dqk], dim=-1)
        q = q.view(b, t, h, dqk).transpose(1, 2)
        k = k.view(b, t, h, dqk).transpose(1, 2)
        vh = v.view(b, t, h, dv).transpose(1, 2)
        attn = (F.silu(q @ k.transpose(-1, -2)) / t).tril()
        o = (attn @ vh).transpose(1, 2).reshape(b, t, h * dv)
        return self.norm(x + self.out(o * F.silu(u)))


class HSTUGenerativeRecommender(nn.Module):
    def __init__(self):
        super().__init__()

        self.embedding_1 = nn.Embedding(200000, 512)
        self.hstuBlock_1 = HSTUBlock(embed_dim=512, num_heads=4, linear_dim=128, attn_dim=128)
        self.hstuBlock_2 = HSTUBlock(embed_dim=512, num_heads=4, linear_dim=128, attn_dim=128)
        self.hstuBlock_3 = HSTUBlock(embed_dim=512, num_heads=4, linear_dim=128, attn_dim=128)
        self.hstuBlock_4 = HSTUBlock(embed_dim=512, num_heads=4, linear_dim=128, attn_dim=128)
        self.rmsNorm_1 = nn.RMSNorm(512)
        self.lmHead_1 = nn.Linear(512, 200000, bias=False)
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md