# Phi-3 Mini Block

> Phi-3 Mini 3.8B decoder block — full MHA (32H, 3072D), SwiGLU FFN (8192), RMSNorm, RoPE

Pick for compact LLMs (≤4B params) when you want modern ingredients (RoPE, SwiGLU) without the full LLaMA-3 footprint.

- Category: NLP/LLM
- Layers: 12
- Parameters: 310.29M
- Input shape (batchless): 1 × 2048
- Output shape: 1 × 2048 × 32064
- Verifier verdict: pass
- Graph JSON: https://neurarch.com/templates/phi3-mini/model.json
- Open on the canvas: https://neurarch.com/?template=phi3-mini

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | tokens | Input | shape=[1, 2048] | 1 × 2048 |
| 2 | embed | Embedding |  | 1 × 2048 × 3072 |
| 3 | rope | RoPE |  |  |
| 4 | norm_attn | RMSNorm | normalizedShape=3072 | 1 × 2048 × 3072 |
| 5 | attn | Grouped Query Attn | embedDim=3072, numHeads=32, numKVHeads=32 | 1 × 2048 × 3072 |
| 6 | residual_1 | Add |  | 1 × 2048 × 3072 |
| 7 | norm_ffn | RMSNorm | normalizedShape=3072 | 1 × 2048 × 3072 |
| 8 | ffn | SwiGLU | embedDim=3072, intermediateSize=8192 | 1 × 2048 × 3072 |
| 9 | residual_2 | Add |  | 1 × 2048 × 3072 |
| 10 | norm_out | RMSNorm | normalizedShape=3072 | 1 × 2048 × 3072 |
| 11 | lm_head | Linear | outFeatures=32064 | 1 × 2048 × 32064 |
| 12 | output | Output |  | 1 × 2048 × 32064 |

## Verifier findings

No finding. Shapes propagate end to end and no advisory rule fires.

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class Phi_3MiniBlock(nn.Module):
    def __init__(self):
        super().__init__()

        self.embedding_1 = nn.Embedding(32064, 3072)
        self.rmsNorm_1 = nn.RMSNorm(3072)
        self.groupedQueryAttention_1 = nn.ModuleDict({
            'q_proj': nn.Linear(3072, 3072,        bias=False),   # 32 heads × 96
            'k_proj': nn.Linear(3072, 3072, bias=False),   # 32 KV heads × 96
            'v_proj': nn.Linear(3072, 3072, bias=False),
            'o_proj': nn.Linear(3072, 3072,        bias=False),
        })  # GQA: 32Q / 32KV heads (requires F.scaled_dot_product_attention)
        self.rmsNorm_2 = nn.RMSNorm(3072)
        self.swiglu_1 = nn.ModuleDict({
            'gate_proj': nn.Linear(3072, 8192, bias=False),
            'up_proj':   nn.Linear(3072, 8192, bias=False),
            'down_proj': nn.Linear(8192, 3072, bias=False),
        })  # SwiGLU FFN (LLaMA-style)
        self.rmsNorm_3 = nn.RMSNorm(3072)
        self.linear_1 = nn.Linear(6291456, 32064)

    def forward(self, x):
        # tokens shape: [1,2048]
        # rope: RoPE applied inside attention (no separate layer needed)
        embedding_embed = self.embedding_1(x)
        rms_norm_m_attn = self.rmsNorm_1(embedding_embed)
        grouped_query_attention_attn = self.groupedQueryAttention_1['o_proj'](F.scaled_dot_product_attention(
            self.groupedQueryAttention_1['q_proj'](rms_norm_m_attn).view(rms_norm_m_attn.size(0),-1,32,96).transpose(1,2),
            self.groupedQueryAttention_1['k_proj'](rms_norm_m_attn).view(rms_norm_m_attn.size(0),-1,32,96).transpose(1,2).repeat_interleave(1,dim=1),
            self.groupedQueryAttention_1['v_proj'](rms_norm_m_attn).view(rms_norm_m_attn.size(0),-1,32,96).transpose(1,2).repeat_interleave(1,dim=1),
            is_causal=True,
        ).transpose(1,2).reshape(rms_norm_m_attn.size(0),-1,3072))
        add_idual1 = grouped_query_attention_attn + embedding_embed
        rms_norm_rm_ffn = self.rmsNorm_2(add_idual1)
        swiglu_ffn = self.swiglu_1['down_proj'](F.silu(self.swiglu_1['gate_proj'](rms_norm_rm_ffn)) * self.swiglu_1['up_proj'](rms_norm_rm_ffn))
        add_idual2 = swiglu_ffn + add_idual1
        rms_norm_rm_out = self.rmsNorm_3(add_idual2)
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md