# tiny-Qwen2ForCausalLM-2.5

Hugging Face: `trl-internal-testing/tiny-Qwen2ForCausalLM-2.5`
Reconstructed from `config.json` alone. No weights were read.

| | |
|---|---|
| Derived from structure | 2,587,048 |
| In the published checkpoint | 2,435,016 (safetensors.total, 2026-09-08) |
| Delta | +6.24% |
| Comparability | comparable |
| Layers | 16 |
| Priced on | A10G (24GB) |
| Est. one run | $2.18 |

## Structure

| # | Layer | Type | Output shape |
|---|---|---|---|
| 1 | Input | Input | 1 × 32768 |
| 2 | Embedding | Embedding | 1 × 32768 × 8 |
| 3 | RoPE | RoPE | 1 × 32768 × 8 |
| 4 | RMSNorm_1_1 | RMSNorm | 1 × 32768 × 8 |
| 5 | Attention_1 | Grouped Query Attn | 1 × 32768 × 8 |
| 6 | Add_1_attn | Add | 1 × 32768 × 8 |
| 7 | RMSNorm_1_2 | RMSNorm | 1 × 32768 × 8 |
| 8 | FFN_1 | SwiGLU | 1 × 32768 × 8 |
| 9 | Add_1_ffn | Add | 1 × 32768 × 8 |
| 10 | RMSNorm_2_1 | RMSNorm | 1 × 32768 × 8 |
| 11 | Attention_2 | Grouped Query Attn | 1 × 32768 × 8 |
| 12 | Add_2_attn | Add | 1 × 32768 × 8 |
| 13 | RMSNorm_2_2 | RMSNorm | 1 × 32768 × 8 |
| 14 | FFN_2 | SwiGLU | 1 × 32768 × 8 |
| 15 | Add_2_ffn | Add | 1 × 32768 × 8 |
| 16 | Final_RMSNorm | RMSNorm | 1 × 32768 × 8 |
| 17 | LM_Head | Linear | 1 × 32768 × 152064 |
| 18 | Output | Output | 1 × 32768 × 152064 |

## Do this to your own model

```
pip install neurarch-trace
neurarch-trace trl-internal-testing/tiny-Qwen2ForCausalLM-2.5 --plan --share
```

- Page: https://neurarch.com/m/tiny-qwen2forcausallm-2-5.html
- All models: https://neurarch.com/m/index.json
- The checks: https://neurarch.com/r/index.json