# Transformer Block

> Transformer encoder block (simplified)

Pick as a generic encoder building block for sequence models when you don't need the modern LLM stack (RoPE/GQA/RMSNorm). Good teaching baseline.

- Category: NLP/LLM
- Layers: 8
- Parameters: 7.09M
- Input shape (batchless): 512 × 768
- Output shape: 512 × 768
- Verifier verdict: warn
- Graph JSON: https://neurarch.com/templates/transformer-block/model.json
- Open on the canvas: https://neurarch.com/?template=transformer-block

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | Input | Input | shape=[512, 768] | 512 × 768 |
| 2 | MultiHeadAttention | Multi-Head Attention | numHeads=8 | 512 × 768 |
| 3 | Add_1 | Add |  | 512 × 768 |
| 4 | LayerNorm_1 | LayerNorm |  | 512 × 768 |
| 5 | FeedForward | Feed Forward | ffDim=3072 | 512 × 768 |
| 6 | Add_2 | Add |  | 512 × 768 |
| 7 | LayerNorm_2 | LayerNorm |  | 512 × 768 |
| 8 | Output | Output |  | 512 × 768 |

## Verifier findings

- **warn** `bn-at-output` at `LayerNorm_2`: "LayerNorm_2" (layerNorm) is the last layer before Output. Normalizing the raw logits constrains the output range and breaks standard loss functions. Fix: Move normalization before the final Linear/Conv layer.
- **warn** `attention-no-pe` at `MultiHeadAttention`: 1 attention layer(s) present but no positional encoding found. Attention is permutation-invariant, without position information the model cannot distinguish token order. Fix: Add a PositionalEncoding (sinusoidal) or RoPE layer before the first attention layer.

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class TransformerBlock(nn.Module):
    def __init__(self):
        super().__init__()

        self.multiHeadAttention_1 = nn.MultiheadAttention(embed_dim=768, num_heads=8, batch_first=True)
        self.layerNorm_1 = nn.LayerNorm(768)
        self.feedForward_1 = nn.Sequential(
            nn.Linear(768, 3072),
            nn.ReLU(),
            nn.Linear(3072, 768)
        )
        self.layerNorm_2 = nn.LayerNorm(768)

    def forward(self, x):
        # Input shape: [512,768]
        multi_head_attention_tion_1 = self.multiHeadAttention_1(x, x, x)[0]
        add_add_1 = multi_head_attention_tion_1
        layer_norm_Norm_1 = self.layerNorm_1(add_add_1)
        feed_forward_ward_1 = self.feedForward_1(layer_norm_Norm_1)
        add_add_2 = feed_forward_ward_1
        layer_norm_Norm_2 = self.layerNorm_2(add_add_2)
        # Output
        return layer_norm_Norm_2


if __name__ == '__main__':
    model = TransformerBlock()
    model.eval()

    x = torch.randn(1, 512, 768)  # (batch, channels, length)
    with torch.no_grad():
        output = model(x)

    print(f'Input  shape : {tuple(x.shape)}')
    print(f'Output shape : {tuple(output.shape)}')
    total = sum(p.numel() for p in model.parameters())
    trainable = sum(p.numel() for p in model.parameters() if p.requires_grad)
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md