# LLaMA-3 Block vs Phi-3 Mini Block

A block from a large model against a whole small one.

**Phi-3 Mini Block has 392M fewer parameters than LLaMA-3 Block: 2 layers added, 6 changed.**

Source: https://neurarch.com/diff/llama3-block-vs-phi3-mini.html

## Sides

| | LLaMA-3 Block | Phi-3 Mini Block |
|---|---|---|
| Layers | 8 | 10 |
| Parameters | 703M | 310M |
| Input | 1 × 2048 | 1 × 2048 |
| Output | 1 × 2048 × 4096 | 1 × 2048 × 32064 |
| Forward-passes | yes | yes |
| Est. train cost | $14.27 | $16.76 |
| T4 16GB | fits | fits |
| A100 40GB | fits | fits |
| H100 80GB | fits | fits |

## Deltas (Phi-3 Mini Block relative to LLaMA-3 Block)

- Parameters: -392M (-55.8%)
- Layers: +2
- Added 2, removed 0, changed 6, unchanged 4

## Layer by layer

| # | Status | LLaMA-3 Block | Params | Output | Phi-3 Mini Block | Params | Output |
|---|---|---|---|---|---|---|---|
| 1 | same | tokens (Input) |  | 1 × 2048 | tokens (Input) |  | 1 × 2048 |
| 2 | changed (dim, maxSeqLen, headDim) | rope (Rope) |  |  | rope (Rope) |  |  |
| 3 | changed (numEmbeddings, embeddingDim) | embed (Embedding) | 525M | 1 × 2048 × 4096 | embed (Embedding) | 99M | 1 × 2048 × 3072 |
| 4 | changed (normalizedShape) | attn_norm (Rms Norm) | 4.1K | 1 × 2048 × 4096 | norm_attn (Rms Norm) | 3.1K | 1 × 2048 × 3072 |
| 5 | changed (embedDim, numKVHeads) | gqa (Grouped Query Attention) | 42M | 1 × 2048 × 4096 | attn (Grouped Query Attention) | 38M | 1 × 2048 × 3072 |
| 6 | same | residual_1 (Add) |  | 1 × 2048 × 4096 | residual_1 (Add) |  | 1 × 2048 × 3072 |
| 7 | changed (normalizedShape) | ffn_norm (Rms Norm) | 4.1K | 1 × 2048 × 4096 | norm_ffn (Rms Norm) | 3.1K | 1 × 2048 × 3072 |
| 8 | changed (embedDim, intermediateSize) | swiglu_ffn (Swiglu) | 135M | 1 × 2048 × 4096 | ffn (Swiglu) | 75M | 1 × 2048 × 3072 |
| 9 | same | residual_2 (Add) |  | 1 × 2048 × 4096 | residual_2 (Add) |  | 1 × 2048 × 3072 |
| 10 | added |  | |  | norm_out (Rms Norm) | 3.1K | 1 × 2048 × 3072 |
| 11 | added |  | |  | lm_head (Linear) |  | 1 × 2048 × 32064 |
| 12 | same | hidden_state (Output) |  | 1 × 2048 × 4096 | output (Output) |  | 1 × 2048 × 32064 |

## What this is not

- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

## Graphs

- LLaMA-3 Block: https://neurarch.com/templates/llama3-block/model.json
- Phi-3 Mini Block: https://neurarch.com/templates/phi3-mini/model.json
- Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`