# T5 Small

> T5 encoder-decoder — bidirectional encoder + masked decoder with cross-attention (512D, 8 heads)

Pick for seq2seq tasks (summarization, translation, QA) where you need both bidirectional understanding and generation in one model.

- Category: NLP/LLM
- Layers: 23
- Parameters: 56.73M
- Input shape (batchless): 1 × 512
- Output shape: 1 × 128 × 32128
- Verifier verdict: warn
- Graph JSON: https://neurarch.com/templates/t5-small/model.json
- Open on the canvas: https://neurarch.com/?template=t5-small

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | encoder_ids | Input | shape=[1, 512] | 1 × 512 |
| 2 | shared_embed | Embedding |  | 1 × 512 × 512 |
| 3 | enc_norm | RMSNorm | normalizedShape=512 | 1 × 512 × 512 |
| 4 | enc_self_attn | Multi-Head Attention | embedDim=512, numHeads=8 | 1 × 512 × 512 |
| 5 | enc_residual | Add |  | 1 × 512 × 512 |
| 6 | enc_ffn_norm | RMSNorm | normalizedShape=512 | 1 × 512 × 512 |
| 7 | enc_ffn | Feed Forward | embedDim=512, ffDim=2048 | 1 × 512 × 512 |
| 8 | enc_ffn_residual | Add |  | 1 × 512 × 512 |
| 9 | enc_out_norm | LayerNorm | normalizedShape=512 | 1 × 512 × 512 |
| 10 | decoder_ids | Input | shape=[1, 128] | 1 × 128 |
| 11 | dec_embed | Embedding |  | 1 × 128 × 512 |
| 12 | dec_sa_norm | RMSNorm | normalizedShape=512 | 1 × 128 × 512 |
| 13 | dec_self_attn | Causal Attention | embedDim=512, numHeads=8 | 1 × 128 × 512 |
| 14 | dec_sa_residual | Add |  | 1 × 128 × 512 |
| 15 | dec_ca_norm | RMSNorm | normalizedShape=512 | 1 × 128 × 512 |
| 16 | cross_attn | Multi-Head Attention | embedDim=512, numHeads=8 | 1 × 128 × 512 |
| 17 | dec_ca_residual | Add |  | 1 × 128 × 512 |
| 18 | dec_ffn_norm | RMSNorm | normalizedShape=512 | 1 × 128 × 512 |
| 19 | dec_ffn | Feed Forward | embedDim=512, ffDim=2048 | 1 × 128 × 512 |
| 20 | dec_ffn_residual | Add |  | 1 × 128 × 512 |
| 21 | dec_out_norm | LayerNorm | normalizedShape=512 | 1 × 128 × 512 |
| 22 | lm_head | Linear | outFeatures=32128 | 1 × 128 × 32128 |
| 23 | logits | Output |  | 1 × 128 × 32128 |

## Verifier findings

- **warn** `attention-no-pe` at `enc_self_attn`: 3 attention layer(s) present but no positional encoding found. Attention is permutation-invariant, without position information the model cannot distinguish token order. Fix: Add a PositionalEncoding (sinusoidal) or RoPE layer before the first attention layer.

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class T5Small(nn.Module):
    def __init__(self):
        super().__init__()

        self.embedding_1 = nn.Embedding(32128, 512)
        self.rmsNorm_1 = nn.RMSNorm(512)
        self.multiHeadAttention_1 = nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)
        self.rmsNorm_2 = nn.RMSNorm(512)
        self.feedForward_1 = nn.Sequential(
            nn.Linear(512, 2048),
            nn.ReLU(),
            nn.Linear(2048, 512)
        )
        self.layerNorm_1 = nn.LayerNorm(512)
        self.embedding_2 = nn.Embedding(32128, 512)
        self.rmsNorm_3 = nn.RMSNorm(512)
        self.causalAttention_1 = nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)
        self.rmsNorm_4 = nn.RMSNorm(512)
        self.multiHeadAttention_2 = nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)
        self.rmsNorm_5 = nn.RMSNorm(512)
        self.feedForward_2 = nn.Sequential(
            nn.Linear(512, 2048),
            nn.ReLU(),
            nn.Linear(2048, 512)
        )
        self.layerNorm_2 = nn.LayerNorm(512)
        self.linear_1 = nn.Linear(512, 32128)

    def forward(self, src, tgt=None):
        # encoder_ids shape: [1,512]
        # decoder_ids shape: [1,128]
        embedding_ng_enc = self.embedding_1(src)
        rms_norm_rm_enc = self.rmsNorm_1(embedding_ng_enc)
        multi_head_attention_ha_enc = self.multiHeadAttention_1(rms_norm_rm_enc, rms_norm_rm_enc, rms_norm_rm_enc)[0]
        add_dd_enc = multi_head_attention_ha_enc
        rms_norm_m_enc2 = self.rmsNorm_2(add_dd_enc)
        feed_forward_rd_enc = self.feedForward_1(rms_norm_m_enc2)
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md