🧾 Behavior Sequence Transformer

Alibaba 2019 BST — user behavior sequence + target → Transformer encoder + concat with user features → MLP → CTR

From Chen et al. (2019). Behavior Sequence Transformer for E-commerce Recommendation in Alibaba. DLP-KDD 2019. This page is the graph, not the PDF: open it, edit it, verify it, export it.

Layers

Parameters

128.81M

Input

Output

Verifier

Clean

Every number on this page is computed from the graph by the same functions the app runs, not written by hand.

Open Behavior Sequence Transformer on the canvas Free, no account needed

When to pick it

Pick for sequential CTR when user behavior history matters (session intent). Production-deployed at Alibaba scale; needs sequence-aware features.

Structure

20 layers. Output shapes are propagated from the input shape, batch dimension excluded.

	Layer	Type	Parameters	Output shape
1	Behavior Seq (T items)	Input	shape=[50]	50
2	Item Embed	Embedding	vocabSize=1000000	50 × 64
3	Pos Encoding	Positional Encoding	maxLen=50	50 × 64
4	Target Item	Input	shape=[1]	1
5	Target Embed	Embedding	vocabSize=1000000	1 × 64
6	[seq; tgt]	Concatenate		51 × 64
7	Transformer Encoder	Transformer Block	numHeads=8, numLayers=1	51 × 64
8	to_channels	Permute		64 × 51
9	Mean Pool	GlobalAvgPool1D		64
10	User Profile	Input	shape=[16]	16
11	User Tower	Linear	outFeatures=32, inFeatures=16	32
12	Concat all	Concatenate		96
13	MLP 1024	Linear	outFeatures=1024, inFeatures=96	1024
14	PReLU	PReLU		1024
15	MLP 512	Linear	outFeatures=512, inFeatures=1024	512
16	PReLU	PReLU		512
17	MLP 256	Linear	outFeatures=256, inFeatures=512	256
18	CTR Head	Linear	outFeatures=1, inFeatures=256	1
19	Sigmoid	Sigmoid		1
20	P(click)	Output		1

What the verifier says

The same 43 single-graph checks that run on every edit in the app, on this graph.

info"Sigmoid" feeds directly into Output. PyTorch's nn.CrossEntropyLoss already applies log-softmax internally, an explicit Softmax causes double-application and degrades training stability. Fix: Remove Softmax/Sigmoid for training. Restore it in a separate inference wrapper or ONNX export. (Sigmoid)
output-activation

infoSigmoid saturates to [0,1] / [-1,1], and its gradient approaches zero for large inputs. In networks deeper than 5 layers, this halts learning in early layers. Fix: Use ReLU, GELU, or SiLU for hidden layers. Keep Sigmoid only at binary classification outputs; Tanh in specific contexts (GAN generators, LSTM gates). (Sigmoid)
vanishing-gradient

info16 layers with no BatchNorm, LayerNorm, or GroupNorm. Without normalization, activations can explode or vanish across layers, causing slow or unstable training. Fix: Add BatchNorm after Conv2d (CV tasks), LayerNorm after attention/FFN (NLP/LLM), or GroupNorm for small batch sizes.
deep-no-norm

info"MLP 256" feeds directly into "CTR Head" with no activation between them. Two stacked linear maps collapse into one (W₂·W₁), so the extra layer costs parameters but adds no representational power. Fix: Add a non-linearity (ReLU/GELU) between them. If this is a deliberate low-rank / factorized projection (down-proj → up-proj), this hint is safe to ignore. (MLP 256)
consecutive-linear-no-activation

infoPyTorch initializes Linear/Conv with Kaiming (He) init, which is derived for ReLU-family activations. Feeding a saturating activation (sigmoid/tanh) from a He-initialized layer starts training in the saturated tails, shrinking early gradients. Fix: Initialize these layers with Xavier instead: nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain("sigmoid"|"tanh")), or switch the activation to a ReLU-family one. (CTR Head)
init-activation-mismatch

The PyTorch it exports

Generated from the graph above. First 46 lines; the app exports the whole file, plus the training loop, the data contract and a deploy bundle.

# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class BehaviorSequenceTransformer(nn.Module):
    def __init__(self):
        super().__init__()

        self.embedding_1 = nn.Embedding(1000000, 64)
        self.embedding_2 = nn.Embedding(1000000, 64)
        self.transformerBlock_1 = nn.TransformerEncoderLayer(d_model=64, nhead=8, dim_feedforward=256, batch_first=True)
        self.linear_1 = nn.Linear(16, 32)
        self.linear_2 = nn.Linear(96, 1024)
        self.prelu_1 = nn.PReLU(num_parameters=1)
        self.linear_3 = nn.Linear(1024, 512)
        self.prelu_2 = nn.PReLU(num_parameters=1)
        self.linear_4 = nn.Linear(512, 256)
        self.linear_5 = nn.Linear(256, 1)

    def forward(self, src, tgt=None, in3=None):
        # Behavior Seq (T items) shape: [50]
        # Target Item shape: [1]
        # User Profile shape: [16]
        embedding_eq_emb = self.embedding_1(src)
        # positionalEncoding: add positional encoding externally (e.g. sinusoidal or learned PE)
        embedding_gt_emb = self.embedding_2(tgt)
        concatenate_at_seq = torch.cat([embedding_eq_emb, embedding_gt_emb], dim=-2)
        transformer_block_trf = self.transformerBlock_1(concatenate_at_seq)
        permute_f_swap = transformer_block_trf.permute(0, 2, 1)
        global_avg_pool1d_f_pool = permute_f_swap.mean(dim=-1)
        linear_ser_fc = self.linear_1(in3)
        concatenate__final = torch.cat([global_avg_pool1d_f_pool, linear_ser_fc], dim=-1)
        linear_mlp1 = self.linear_2(concatenate__final)
        prelu_act1 = self.prelu_1(linear_mlp1)
        linear_mlp2 = self.linear_3(prelu_act1)
        prelu_act2 = self.prelu_2(linear_mlp2)
        linear_mlp3 = self.linear_4(prelu_act2)
        linear_ctr = self.linear_5(linear_mlp3)
        sigmoid_sig = torch.sigmoid(linear_ctr)
        # Output
        return sigmoid_sig

For agents

This architecture is machine-readable end to end. An agent can list the set, fetch this graph, edit it, and have the edit verified before any GPU time is spent.

Markdown twin Index JSON This graph as JSON MCP server POST /api/v1/check

Also in Recommendation

🗼 Two-Tower

User+Item dual encoder for retrieval — embeddings → MLP per side → dot product score

Meta's Deep Learning Recommendation Model — bottom MLP for dense, embedding for sparse, feature interaction, top MLP

14 layers · 64.35M

🤝 NeuMF (Fused GMF + MLP)

He et al

16 layers · 105.61M