# BERT Base

> BERT-Base encoder — bidirectional MHA (12 heads, 768D), no causal mask, 30K vocab

Pick for text classification / NLU with limited labels — pretrained encoder + small head fine-tunes reliably. 512-token cap.

- Category: NLP/LLM
- Layers: 11
- Parameters: 31.12M
- Input shape (batchless): 1 × 512
- Output shape: 1 × 512 × 768
- Verifier verdict: pass
- Graph JSON: https://neurarch.com/templates/bert-base/model.json
- Open on the canvas: https://neurarch.com/?template=bert-base

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | input_ids | Input | shape=[1, 512] | 1 × 512 |
| 2 | word_embed | Embedding |  | 1 × 512 × 768 |
| 3 | pos_embed | Positional Encoding | embedDim=768, maxLen=512 | 1 × 512 × 768 |
| 4 | embed_norm | LayerNorm | normalizedShape=768 | 1 × 512 × 768 |
| 5 | embed_drop | Dropout | p=0.1 | 1 × 512 × 768 |
| 6 | self_attn | Multi-Head Attention | embedDim=768, numHeads=12 | 1 × 512 × 768 |
| 7 | norm | LayerNorm | normalizedShape=768 | 1 × 512 × 768 |
| 8 | dense | Feed Forward | embedDim=768, ffDim=3072 | 1 × 512 × 768 |
| 9 | norm | LayerNorm | normalizedShape=768 | 1 × 512 × 768 |
| 10 | dense | Linear | outFeatures=768, inFeatures=768 | 1 × 512 × 768 |
| 11 | cls_embedding | Output |  | 1 × 512 × 768 |

## Verifier findings

No finding. Shapes propagate end to end and no advisory rule fires.

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class BERTBase(nn.Module):
    def __init__(self):
        super().__init__()

        self.embedding_1 = nn.Embedding(30522, 768)
        self.layerNorm_1 = nn.LayerNorm(768)
        self.dropout_1 = nn.Dropout(p=0.1)
        self.multiHeadAttention_1 = nn.MultiheadAttention(embed_dim=768, num_heads=12, batch_first=True)
        self.layerNorm_2 = nn.LayerNorm(768)
        self.feedForward_1 = nn.Sequential(
            nn.Linear(768, 3072),
            nn.ReLU(),
            nn.Linear(3072, 768)
        )
        self.layerNorm_3 = nn.LayerNorm(768)
        self.linear_1 = nn.Linear(768, 768)

    def forward(self, x):
        # input_ids shape: [1,512]
        embedding_ng_tok = self.embedding_1(x)
        # positionalEncoding: add positional encoding externally (e.g. sinusoidal or learned PE)
        layer_norm_rm_emb = self.layerNorm_1(embedding_ng_tok)
        dropout_ut_emb = self.dropout_1(layer_norm_rm_emb)
        multi_head_attention_mha_1 = self.multiHeadAttention_1(dropout_ut_emb, dropout_ut_emb, dropout_ut_emb)[0]
        layer_norm_Norm_1 = self.layerNorm_2(multi_head_attention_mha_1)
        feed_forward_ward_1 = self.feedForward_1(layer_norm_Norm_1)
        layer_norm_Norm_2 = self.layerNorm_3(feed_forward_ward_1)
        linear_r_pool = self.linear_1(layer_norm_Norm_2)
        # Output
        return linear_r_pool


if __name__ == '__main__':
    model = BERTBase()
    model.eval()

    x = torch.randint(0, 50000, (1, 512))  # (batch, features)
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md