# t5-small

Hugging Face: `google-t5/t5-small`
Reconstructed from `config.json` alone. No weights were read.

| | |
|---|---|
| Derived from structure | 60,576,000 |
| In the published checkpoint | 60,506,880 (safetensors.total, 2026-09-06) |
| Delta | +0.11% |
| Comparability | comparable |
| Layers | 70 |
| Priced on | A10G (24GB) |
| Est. one run | $1.14 |

## Structure

| # | Layer | Type | Output shape |
|---|---|---|---|
| 1 | Input | Input | 1 × 512 |
| 2 | Encoder_Embedding | Embedding | 1 × 512 × 512 |
| 3 | Relative_Position_Bias | Relative Position Bias | 1 × 512 × 512 |
| 4 | Enc_Attn_1 | Multi-Head Attention | 1 × 512 × 512 |
| 5 | Enc_Add_1 | Add | 1 × 512 × 512 |
| 6 | Enc_Norm_1 | LayerNorm | 1 × 512 × 512 |
| 7 | Enc_FFN_1 | Feed Forward | 1 × 512 × 512 |
| 8 | Enc_Attn_2 | Multi-Head Attention | 1 × 512 × 512 |
| 9 | Enc_Add_2 | Add | 1 × 512 × 512 |
| 10 | Enc_Norm_2 | LayerNorm | 1 × 512 × 512 |
| 11 | Enc_FFN_2 | Feed Forward | 1 × 512 × 512 |
| 12 | Enc_Attn_3 | Multi-Head Attention | 1 × 512 × 512 |
| 13 | Enc_Add_3 | Add | 1 × 512 × 512 |
| 14 | Enc_Norm_3 | LayerNorm | 1 × 512 × 512 |
| 15 | Enc_FFN_3 | Feed Forward | 1 × 512 × 512 |
| 16 | Enc_Attn_4 | Multi-Head Attention | 1 × 512 × 512 |
| 17 | Enc_Add_4 | Add | 1 × 512 × 512 |
| 18 | Enc_Norm_4 | LayerNorm | 1 × 512 × 512 |
| 19 | Enc_FFN_4 | Feed Forward | 1 × 512 × 512 |
| 20 | Enc_Attn_5 | Multi-Head Attention | 1 × 512 × 512 |
| 21 | Enc_Add_5 | Add | 1 × 512 × 512 |
| 22 | Enc_Norm_5 | LayerNorm | 1 × 512 × 512 |
| 23 | Enc_FFN_5 | Feed Forward | 1 × 512 × 512 |
| 24 | Enc_Attn_6 | Multi-Head Attention | 1 × 512 × 512 |
| 25 | Enc_Add_6 | Add | 1 × 512 × 512 |
| 26 | Enc_Norm_6 | LayerNorm | 1 × 512 × 512 |
| 27 | Enc_FFN_6 | Feed Forward | 1 × 512 × 512 |
| 28 | Decoder_Embedding | Embedding | 1 × 512 × 512 |
| 29 | Dec_SelfAttn_1 | Multi-Head Attention | 1 × 512 × 512 |
| 30 | Dec_Add_1 | Add | 1 × 512 × 512 |
| 31 | Dec_Norm1_1 | LayerNorm | 1 × 512 × 512 |
| 32 | Dec_CrossAttn_1 | Cross-Attention | 1 × 512 × 512 |
| 33 | Dec_Add2_1 | Add | 1 × 512 × 512 |
| 34 | Dec_Norm2_1 | LayerNorm | 1 × 512 × 512 |
| 35 | Dec_FFN_1 | Feed Forward | 1 × 512 × 512 |
| 36 | Dec_SelfAttn_2 | Multi-Head Attention | 1 × 512 × 512 |
| 37 | Dec_Add_2 | Add | 1 × 512 × 512 |
| 38 | Dec_Norm1_2 | LayerNorm | 1 × 512 × 512 |
| 39 | Dec_CrossAttn_2 | Cross-Attention | 1 × 512 × 512 |
| 40 | Dec_Add2_2 | Add | 1 × 512 × 512 |
| 41 | Dec_Norm2_2 | LayerNorm | 1 × 512 × 512 |
| 42 | Dec_FFN_2 | Feed Forward | 1 × 512 × 512 |
| 43 | Dec_SelfAttn_3 | Multi-Head Attention | 1 × 512 × 512 |
| 44 | Dec_Add_3 | Add | 1 × 512 × 512 |
| 45 | Dec_Norm1_3 | LayerNorm | 1 × 512 × 512 |
| 46 | Dec_CrossAttn_3 | Cross-Attention | 1 × 512 × 512 |
| 47 | Dec_Add2_3 | Add | 1 × 512 × 512 |
| 48 | Dec_Norm2_3 | LayerNorm | 1 × 512 × 512 |
| 49 | Dec_FFN_3 | Feed Forward | 1 × 512 × 512 |
| 50 | Dec_SelfAttn_4 | Multi-Head Attention | 1 × 512 × 512 |
| 51 | Dec_Add_4 | Add | 1 × 512 × 512 |
| 52 | Dec_Norm1_4 | LayerNorm | 1 × 512 × 512 |
| 53 | Dec_CrossAttn_4 | Cross-Attention | 1 × 512 × 512 |
| 54 | Dec_Add2_4 | Add | 1 × 512 × 512 |
| 55 | Dec_Norm2_4 | LayerNorm | 1 × 512 × 512 |
| 56 | Dec_FFN_4 | Feed Forward | 1 × 512 × 512 |
| 57 | Dec_SelfAttn_5 | Multi-Head Attention | 1 × 512 × 512 |
| 58 | Dec_Add_5 | Add | 1 × 512 × 512 |
| 59 | Dec_Norm1_5 | LayerNorm | 1 × 512 × 512 |
| 60 | Dec_CrossAttn_5 | Cross-Attention | 1 × 512 × 512 |
| 61 | Dec_Add2_5 | Add | 1 × 512 × 512 |
| 62 | Dec_Norm2_5 | LayerNorm | 1 × 512 × 512 |
| 63 | Dec_FFN_5 | Feed Forward | 1 × 512 × 512 |
| 64 | Dec_SelfAttn_6 | Multi-Head Attention | 1 × 512 × 512 |
| 65 | Dec_Add_6 | Add | 1 × 512 × 512 |
| 66 | Dec_Norm1_6 | LayerNorm | 1 × 512 × 512 |
| 67 | Dec_CrossAttn_6 | Cross-Attention | 1 × 512 × 512 |
| 68 | Dec_Add2_6 | Add | 1 × 512 × 512 |
| 69 | Dec_Norm2_6 | LayerNorm | 1 × 512 × 512 |
| 70 | Dec_FFN_6 | Feed Forward | 1 × 512 × 512 |
| 71 | LM_Head | Linear | 1 × 512 × 32128 |
| 72 | Output | Output | 1 × 512 × 32128 |

## Do this to your own model

```
pip install neurarch-trace
neurarch-trace google-t5/t5-small --plan --share
```

- Page: https://neurarch.com/m/t5-small.html
- All models: https://neurarch.com/m/index.json
- The checks: https://neurarch.com/r/index.json