Architectures / Recommendation
🧾 Behavior Sequence Transformer
Alibaba 2019 BST — user behavior sequence + target → Transformer encoder + concat with user features → MLP → CTR
Layers
20
Parameters
128.76M
Input
50
Output
1
Verifier
Clean
Every number on this page is computed from the graph by the same functions the app runs, not written by hand.
Open Behavior Sequence Transformer on the canvas
Free, no account needed
When to pick it
Pick for sequential CTR when user behavior history matters (session intent). Production-deployed at Alibaba scale; needs sequence-aware features.
Structure
20 layers. Output shapes are propagated from the input shape, batch dimension excluded.
| Layer | Type | Parameters | Output shape | |
|---|---|---|---|---|
| 1 | Behavior Seq (T items) | Input | shape=[50] | 50 |
| 2 | Item Embed | Embedding | vocabSize=1000000 | 50 × 64 |
| 3 | Pos Encoding | Positional Encoding | maxLen=50 | 50 × 64 |
| 4 | Target Item | Input | shape=[1] | 1 |
| 5 | Target Embed | Embedding | vocabSize=1000000 | 1 × 64 |
| 6 | [seq; tgt] | Concatenate | 51 × 64 | |
| 7 | Transformer Encoder | Transformer Block | numHeads=8, numLayers=1 | 51 × 64 |
| 8 | to_channels | Permute | 64 × 51 | |
| 9 | Mean Pool | GlobalAvgPool1D | 64 | |
| 10 | User Profile | Input | shape=[16] | 16 |
| 11 | User Tower | Linear | outFeatures=32, inFeatures=16 | 32 |
| 12 | Concat all | Concatenate | 96 | |
| 13 | MLP 1024 | Linear | outFeatures=1024, inFeatures=96 | 1024 |
| 14 | PReLU | PReLU | 1024 | |
| 15 | MLP 512 | Linear | outFeatures=512, inFeatures=1024 | 512 |
| 16 | PReLU | PReLU | 512 | |
| 17 | MLP 256 | Linear | outFeatures=256, inFeatures=512 | 256 |
| 18 | CTR Head | Linear | outFeatures=1, inFeatures=256 | 1 |
| 19 | Sigmoid | Sigmoid | 1 | |
| 20 | P(click) | Output | 1 |
What the verifier says
The same 41 structural checks that run on every edit in the app, on this graph.
info"Sigmoid" feeds directly into Output. PyTorch's nn.CrossEntropyLoss already applies log-softmax internally, an explicit Softmax causes double-application and degrades training stability. Fix: Remove Softmax/Sigmoid for training. Restore it in a separate inference wrapper or ONNX export. (Sigmoid)
output-activation
output-activation
infoSigmoid saturates to [0,1] / [-1,1], and its gradient approaches zero for large inputs. In networks deeper than 5 layers, this halts learning in early layers. Fix: Use ReLU, GELU, or SiLU for hidden layers. Keep Sigmoid only at binary classification outputs; Tanh in specific contexts (GAN generators, LSTM gates). (Sigmoid)
vanishing-gradient
vanishing-gradient
info16 layers with no BatchNorm, LayerNorm, or GroupNorm. Without normalization, activations can explode or vanish across layers, causing slow or unstable training. Fix: Add BatchNorm after Conv2d (CV tasks), LayerNorm after attention/FFN (NLP/LLM), or GroupNorm for small batch sizes.
deep-no-norm
deep-no-norm
info"MLP 256" feeds directly into "CTR Head" with no activation between them. Two stacked linear maps collapse into one (W₂·W₁), so the extra layer costs parameters but adds no representational power. Fix: Add a non-linearity (ReLU/GELU) between them. If this is a deliberate low-rank / factorized projection (down-proj → up-proj), this hint is safe to ignore. (MLP 256)
consecutive-linear-no-activation
consecutive-linear-no-activation
infoPyTorch initializes Linear/Conv with Kaiming (He) init, which is derived for ReLU-family activations. Feeding a saturating activation (sigmoid/tanh) from a He-initialized layer starts training in the saturated tails, shrinking early gradients. Fix: Initialize these layers with Xavier instead: nn.init.xavier_uniform_(w, gain=nn.init.calculate_gain("sigmoid"|"tanh")), or switch the activation to a ReLU-family one. (CTR Head)
init-activation-mismatch
init-activation-mismatch
The PyTorch it exports
Generated from the graph above. First 46 lines; the app exports the whole file, plus the training loop, the data contract and a deploy bundle.
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision (usually pre-installed)
#
# WARNING: 2 layer(s) below are not yet supported by the PyTorch
# exporter and pass their input through UNCHANGED in forward():
# - to_channels (permute)
# - Mean Pool (globalAvgPool1d)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple
class BehaviorSequenceTransformer(nn.Module):
def __init__(self):
super().__init__()
self.embedding_1 = nn.Embedding(1000000, 64)
self.embedding_2 = nn.Embedding(1000000, 64)
self.transformerBlock_1 = nn.TransformerEncoderLayer(d_model=512, nhead=8, dim_feedforward=2048, batch_first=True)
self.linear_1 = nn.Linear(16, 32)
self.linear_2 = nn.Linear(96, 1024)
self.prelu_1 = nn.PReLU(num_parameters=1)
self.linear_3 = nn.Linear(1024, 512)
self.prelu_2 = nn.PReLU(num_parameters=1)
self.linear_4 = nn.Linear(512, 256)
self.linear_5 = nn.Linear(256, 1)
def forward(self, src, tgt=None):
# Behavior Seq (T items) shape: [50]
# Target Item shape: [1]
# User Profile shape: [16]
embedding_eq_emb = self.embedding_1(src)
# positionalEncoding: add positional encoding externally (e.g. sinusoidal or learned PE)
embedding_gt_emb = self.embedding_2(tgt)
concatenate_at_seq = torch.cat([embedding_eq_emb, embedding_gt_emb], dim=0)
transformer_block_trf = self.transformerBlock_1(concatenate_at_seq)
# TODO: layer 'to_channels' (permute) is not yet supported by the exporter; passing through unchanged
# TODO: layer 'Mean Pool' (globalAvgPool1d) is not yet supported by the exporter; passing through unchanged
linear_ser_fc = self.linear_1(x)
concatenate__final = torch.cat([transformer_block_trf, linear_ser_fc], dim=-1)
linear_mlp1 = self.linear_2(concatenate__final)
prelu_act1 = self.prelu_1(linear_mlp1)
linear_mlp2 = self.linear_3(prelu_act1)
prelu_act2 = self.prelu_2(linear_mlp2)For agents
This architecture is machine-readable end to end. An agent can list the set, fetch this graph, edit it, and have the edit verified before any GPU time is spent.