# LLaMA-3 Block

> LLaMA-3 decoder block — GQA (32H/8KV), SwiGLU FFN, RMSNorm, residual streams

Pick when building a modern dense LLM (7B–70B class). Strongest open recipe per param; GQA cuts KV cache, RoPE handles long context.

- Category: NLP/LLM
- Layers: 10
- Parameters: 702.55M
- Input shape (batchless): 1 × 2048
- Output shape: 1 × 2048 × 4096
- Verifier verdict: pass
- Graph JSON: https://neurarch.com/templates/llama3-block/model.json
- Open on the canvas: https://neurarch.com/?template=llama3-block

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | tokens | Input | shape=[1, 2048] | 1 × 2048 |
| 2 | embed | Embedding |  | 1 × 2048 × 4096 |
| 3 | attn_norm | RMSNorm | normalizedShape=4096 | 1 × 2048 × 4096 |
| 4 | gqa | Grouped Query Attn | embedDim=4096, numHeads=32, numKVHeads=8 | 1 × 2048 × 4096 |
| 5 | rope | RoPE |  |  |
| 6 | residual_1 | Add |  | 1 × 2048 × 4096 |
| 7 | ffn_norm | RMSNorm | normalizedShape=4096 | 1 × 2048 × 4096 |
| 8 | swiglu_ffn | SwiGLU | embedDim=4096, intermediateSize=11008 | 1 × 2048 × 4096 |
| 9 | residual_2 | Add |  | 1 × 2048 × 4096 |
| 10 | hidden_state | Output |  | 1 × 2048 × 4096 |

## Verifier findings

No finding. Shapes propagate end to end and no advisory rule fires.

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class LLaMA_3DecoderBlock(nn.Module):
    def __init__(self):
        super().__init__()

        self.embedding_1 = nn.Embedding(128256, 4096)
        self.rmsNorm_1 = nn.RMSNorm(4096)
        self.groupedQueryAttention_1 = nn.ModuleDict({
            'q_proj': nn.Linear(4096, 4096,        bias=False),   # 32 heads × 128
            'k_proj': nn.Linear(4096, 1024, bias=False),   # 8 KV heads × 128
            'v_proj': nn.Linear(4096, 1024, bias=False),
            'o_proj': nn.Linear(4096, 4096,        bias=False),
        })  # GQA: 32Q / 8KV heads (requires F.scaled_dot_product_attention)
        self.rmsNorm_2 = nn.RMSNorm(4096)
        self.swiglu_1 = nn.ModuleDict({
            'gate_proj': nn.Linear(4096, 11008, bias=False),
            'up_proj':   nn.Linear(4096, 11008, bias=False),
            'down_proj': nn.Linear(11008, 4096, bias=False),
        })  # SwiGLU FFN (LLaMA-style)

    def forward(self, x):
        # tokens shape: [1,2048]
        # rope: RoPE applied inside attention (no separate layer needed)
        embedding_ding_1 = self.embedding_1(x)
        rms_norm_Norm_1 = self.rmsNorm_1(embedding_ding_1)
        grouped_query_attention_tion_1 = self.groupedQueryAttention_1['o_proj'](F.scaled_dot_product_attention(
            self.groupedQueryAttention_1['q_proj'](rms_norm_Norm_1).view(rms_norm_Norm_1.size(0),-1,32,128).transpose(1,2),
            self.groupedQueryAttention_1['k_proj'](rms_norm_Norm_1).view(rms_norm_Norm_1.size(0),-1,8,128).transpose(1,2).repeat_interleave(4,dim=1),
            self.groupedQueryAttention_1['v_proj'](rms_norm_Norm_1).view(rms_norm_Norm_1.size(0),-1,8,128).transpose(1,2).repeat_interleave(4,dim=1),
            is_causal=True,
        ).transpose(1,2).reshape(rms_norm_Norm_1.size(0),-1,4096))
        add_add_1 = grouped_query_attention_tion_1 + embedding_ding_1
        rms_norm_Norm_2 = self.rmsNorm_2(add_add_1)
        swiglu_iglu_1 = self.swiglu_1['down_proj'](F.silu(self.swiglu_1['gate_proj'](rms_norm_Norm_2)) * self.swiglu_1['up_proj'](rms_norm_Norm_2))
        add_add_2 = swiglu_iglu_1 + add_add_1
        # Output
        return add_add_2
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md