# Transformer Block vs LLaMA-3 Block

What changed in the transformer block between 2017 and now.

**LLaMA-3 Block has 695M more parameters than Transformer Block: 5 layers added, 3 removed, 2 changed.**

Source: https://neurarch.com/diff/transformer-block-vs-llama3-block.html

## Sides

| | Transformer Block | LLaMA-3 Block |
|---|---|---|
| Layers | 6 | 8 |
| Parameters | 7.1M | 703M |
| Input | 512 × 768 | 1 × 2048 |
| Output | 512 × 768 | 1 × 2048 × 4096 |
| Forward-passes | yes | yes |
| Est. train cost | $0.185 | $14.27 |
| T4 16GB | fits | fits |
| A100 40GB | fits | fits |
| H100 80GB | fits | fits |

## Deltas (LLaMA-3 Block relative to Transformer Block)

- Parameters: +695M (99× the size)
- Layers: +2
- Added 5, removed 3, changed 2, unchanged 3

## Layer by layer

| # | Status | Transformer Block | Params | Output | LLaMA-3 Block | Params | Output |
|---|---|---|---|---|---|---|---|
| 1 | changed (shape) | Input (Input) |  | 512 × 768 | tokens (Input) |  | 1 × 2048 |
| 2 | removed | MultiHeadAttention (Multi Head Attention) | 2.4M | 512 × 768 |  | |  |
| 3 | added |  | |  | rope (Rope) |  |  |
| 4 | added |  | |  | embed (Embedding) | 525M | 1 × 2048 × 4096 |
| 5 | added |  | |  | attn_norm (Rms Norm) | 4.1K | 1 × 2048 × 4096 |
| 6 | added |  | |  | gqa (Grouped Query Attention) | 42M | 1 × 2048 × 4096 |
| 7 | same | Add_1 (Add) |  | 512 × 768 | residual_1 (Add) |  | 1 × 2048 × 4096 |
| 8 | changed (type, normalizedShape) | LayerNorm_1 (Layer Norm) |  | 512 × 768 | ffn_norm (Rms Norm) | 4.1K | 1 × 2048 × 4096 |
| 9 | removed | FeedForward (Feed Forward) | 4.7M | 512 × 768 |  | |  |
| 10 | added |  | |  | swiglu_ffn (Swiglu) | 135M | 1 × 2048 × 4096 |
| 11 | same | Add_2 (Add) |  | 512 × 768 | residual_2 (Add) |  | 1 × 2048 × 4096 |
| 12 | removed | LayerNorm_2 (Layer Norm) |  | 512 × 768 |  | |  |
| 13 | same | Output (Output) |  | 512 × 768 | hidden_state (Output) |  | 1 × 2048 × 4096 |

## What this is not

- The two are priced at different declared inputs (512 × 768 against 1 × 2048), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

## Graphs

- Transformer Block: https://neurarch.com/templates/transformer-block/model.json
- LLaMA-3 Block: https://neurarch.com/templates/llama3-block/model.json
- Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`