Architectures / NLP/LLM
🤖 Transformer Block
Transformer encoder block (simplified)
Layers
8
Parameters
7.09M
Input
512 × 768
Output
512 × 768
Verifier
2 advisories
Every number on this page is computed from the graph by the same functions the app runs, not written by hand.
Open Transformer Block on the canvas
Free, no account needed
When to pick it
Pick as a generic encoder building block for sequence models when you don't need the modern LLM stack (RoPE/GQA/RMSNorm). Good teaching baseline.
Structure
8 layers. Output shapes are propagated from the input shape, batch dimension excluded.
| Layer | Type | Parameters | Output shape | |
|---|---|---|---|---|
| 1 | Input | Input | shape=[512, 768] | 512 × 768 |
| 2 | MultiHeadAttention | Multi-Head Attention | numHeads=8 | 512 × 768 |
| 3 | Add_1 | Add | 512 × 768 | |
| 4 | LayerNorm_1 | LayerNorm | 512 × 768 | |
| 5 | FeedForward | Feed Forward | ffDim=3072 | 512 × 768 |
| 6 | Add_2 | Add | 512 × 768 | |
| 7 | LayerNorm_2 | LayerNorm | 512 × 768 | |
| 8 | Output | Output | 512 × 768 |
What the verifier says
The same 41 structural checks that run on every edit in the app, on this graph.
warn"LayerNorm_2" (layerNorm) is the last layer before Output. Normalizing the raw logits constrains the output range and breaks standard loss functions. Fix: Move normalization before the final Linear/Conv layer. (LayerNorm_2)
bn-at-output
bn-at-output
warn1 attention layer(s) present but no positional encoding found. Attention is permutation-invariant, without position information the model cannot distinguish token order. Fix: Add a PositionalEncoding (sinusoidal) or RoPE layer before the first attention layer. (MultiHeadAttention)
attention-no-pe
attention-no-pe
The PyTorch it exports
Generated from the graph above. First 46 lines; the app exports the whole file, plus the training loop, the data contract and a deploy bundle.
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision (usually pre-installed)
import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple
class TransformerBlock(nn.Module):
def __init__(self):
super().__init__()
self.multiHeadAttention_1 = nn.MultiheadAttention(embed_dim=768, num_heads=8, batch_first=True)
self.layerNorm_1 = nn.LayerNorm(768)
self.feedForward_1 = nn.Sequential(
nn.Linear(768, 3072),
nn.ReLU(),
nn.Linear(3072, 768)
)
self.layerNorm_2 = nn.LayerNorm(768)
def forward(self, x):
# Input shape: [512,768]
multi_head_attention_tion_1 = self.multiHeadAttention_1(x, x, x)[0]
add_add_1 = multi_head_attention_tion_1
layer_norm_Norm_1 = self.layerNorm_1(add_add_1)
feed_forward_ward_1 = self.feedForward_1(layer_norm_Norm_1)
add_add_2 = feed_forward_ward_1
layer_norm_Norm_2 = self.layerNorm_2(add_add_2)
# Output
return layer_norm_Norm_2
if __name__ == '__main__':
model = TransformerBlock()
model.eval()
x = torch.randn(1, 512, 768) # (batch, channels, length)
with torch.no_grad():
output = model(x)
print(f'Input shape : {tuple(x.shape)}')
print(f'Output shape : {tuple(output.shape)}')
total = sum(p.numel() for p in model.parameters())
trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)For agents
This architecture is machine-readable end to end. An agent can list the set, fetch this graph, edit it, and have the edit verified before any GPU time is spent.