# GPT-2

> GPT-2 Small — causal transformer block (768D, 12 heads, 4× FFN)

Pick for single-GPU language modeling experiments and as a teaching reference for the canonical decoder-only stack. Modern LLMs prefer LLaMA-3 / Phi-3 blocks.

- Category: NLP/LLM
- Layers: 12
- Parameters: 84.33M
- Input shape (batchless): 1 × 1024
- Output shape: 1 × 1024 × 50257
- Verifier verdict: pass
- Graph JSON: https://neurarch.com/templates/gpt2/model.json
- Open on the canvas: https://neurarch.com/?template=gpt2

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | tokens | Input | shape=[1, 1024] | 1 × 1024 |
| 2 | token_embed | Embedding |  | 1 × 1024 × 768 |
| 3 | pos_embed | Positional Encoding | embedDim=768, maxLen=1024 | 1 × 1024 × 768 |
| 4 | ln_1 | LayerNorm | normalizedShape=768 | 1 × 1024 × 768 |
| 5 | attn | Causal Attention | embedDim=768, numHeads=12 | 1 × 1024 × 768 |
| 6 | residual_1 | Add |  | 1 × 1024 × 768 |
| 7 | ln_2 | LayerNorm | normalizedShape=768 | 1 × 1024 × 768 |
| 8 | mlp | Feed Forward | ffDim=3072 | 1 × 1024 × 768 |
| 9 | residual_2 | Add |  | 1 × 1024 × 768 |
| 10 | ln_f | LayerNorm | normalizedShape=768 | 1 × 1024 × 768 |
| 11 | lm_head | Linear | outFeatures=50257 | 1 × 1024 × 50257 |
| 12 | logits | Output |  | 1 × 1024 × 50257 |

## Verifier findings

No finding. Shapes propagate end to end and no advisory rule fires.

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class GPT_2Block(nn.Module):
    def __init__(self):
        super().__init__()

        self.embedding_1 = nn.Embedding(50257, 768)
        self.layerNorm_1 = nn.LayerNorm(768)
        self.causalAttention_1 = nn.MultiheadAttention(embed_dim=768, num_heads=12, batch_first=True)
        self.layerNorm_2 = nn.LayerNorm(768)
        self.feedForward_1 = nn.Sequential(
            nn.Linear(768, 3072),
            nn.ReLU(),
            nn.Linear(3072, 768)
        )
        self.layerNorm_3 = nn.LayerNorm(768)
        self.linear_1 = nn.Linear(768, 50257)

    def forward(self, x):
        # tokens shape: [1,1024]
        embedding_ding_1 = self.embedding_1(x)
        # positionalEncoding: add positional encoding externally (e.g. sinusoidal or learned PE)
        layer_norm_Norm_1 = self.layerNorm_1(embedding_ding_1)
        causal_attention_tion_1 = self.causalAttention_1(layer_norm_Norm_1, layer_norm_Norm_1, layer_norm_Norm_1)[0]
        add_add_1 = causal_attention_tion_1 + embedding_ding_1
        layer_norm_Norm_2 = self.layerNorm_2(add_add_1)
        feed_forward_ward_1 = self.feedForward_1(layer_norm_Norm_2)
        add_add_2 = feed_forward_ward_1 + add_add_1
        layer_norm_Norm_3 = self.layerNorm_3(add_add_2)
        linear_near_1 = self.linear_1(layer_norm_Norm_3)
        # Output
        return linear_near_1


if __name__ == '__main__':
    model = GPT_2Block()
    model.eval()

    x = torch.randint(0, 50000, (1, 1024))  # (batch, features)
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md