# LLaMA-3 Block vs Mixtral MoE Block

A dense feed-forward against a mixture of experts, at the block level.

**Mixtral MoE Block has 749M more parameters than LLaMA-3 Block: 1 layer added, 2 removed, 2 changed.**

Source: https://neurarch.com/diff/llama3-block-vs-mixtral-block.html

## Sides

| | LLaMA-3 Block | Mixtral MoE Block |
|---|---|---|
| Layers | 8 | 7 |
| Parameters | 703M | 1.5B |
| Input | 1 × 2048 | 1 × 4096 × 4096 |
| Output | 1 × 2048 × 4096 | 1 × 4096 × 4096 |
| Forward-passes | yes | yes |
| Est. train cost | $14.27 | $63.07 |
| T4 16GB | fits | no |
| A100 40GB | fits | fits |
| H100 80GB | fits | fits |

## Deltas (Mixtral MoE Block relative to LLaMA-3 Block)

- Parameters: +749M (+107%)
- Layers: -1
- Added 1, removed 2, changed 2, unchanged 6

## Layer by layer

| # | Status | LLaMA-3 Block | Params | Output | Mixtral MoE Block | Params | Output |
|---|---|---|---|---|---|---|---|
| 1 | changed (shape) | tokens (Input) |  | 1 × 2048 | hidden_states (Input) |  | 1 × 4096 × 4096 |
| 2 | changed (maxSeqLen) | rope (Rope) |  |  | rotary_emb (Rope) |  |  |
| 3 | removed | embed (Embedding) | 525M | 1 × 2048 × 4096 |  | |  |
| 4-7 | same | 4 unchanged layers | | | | | |
| 8 | removed | swiglu_ffn (Swiglu) | 135M | 1 × 2048 × 4096 |  | |  |
| 9 | added |  | |  | block_sparse_moe (Moe Layer) | 1.4B | 1 × 4096 × 4096 |
| 10 | same | residual_2 (Add) |  | 1 × 2048 × 4096 | moe_residual (Add) |  | 1 × 4096 × 4096 |
| 11 | same | hidden_state (Output) |  | 1 × 2048 × 4096 | hidden_out (Output) |  | 1 × 4096 × 4096 |

## What this is not

- The two are priced at different declared inputs (1 × 2048 against 1 × 4096 × 4096), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

## Graphs

- LLaMA-3 Block: https://neurarch.com/templates/llama3-block/model.json
- Mixtral MoE Block: https://neurarch.com/templates/mixtral-block/model.json
- Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`