# TimeMoE-50M

Hugging Face: `Maple728/TimeMoE-50M`
Reconstructed from `config.json` alone. No weights were read.

| | |
|---|---|
| Derived from structure | 34,577,664 |
| In the published checkpoint | 113,352,192 (safetensors.total, 2024-10-22) |
| Delta | -69.5% |
| Comparability | custom-code |
| Layers | 50 |
| Priced on | A10G (24GB) |
| Est. one run | $6.19 |

> This repository ships its own modeling code (`auto_map`, e.g. `configuration_time_moe.py`), so `config.json` names a class in the repo rather than an architecture `transformers` defines. The graph below is what those config keys mean under `transformers` semantics, which is not necessarily what the repo's own file builds. A gap here is a statement about what we read, not about the checkpoint.

## Structure

| # | Layer | Type | Output shape |
|---|---|---|---|
| 1 | Input | Input | 1 × 4096 |
| 2 | Embedding | Embedding | 1 × 4096 × 384 |
| 3 | Positional_Embedding | Learned Pos Embed | 1 × 4096 × 384 |
| 4 | Attention_1 | Multi-Head Attention | 1 × 4096 × 384 |
| 5 | Add_1 | Add | 1 × 4096 × 384 |
| 6 | LayerNorm_1_1 | LayerNorm | 1 × 4096 × 384 |
| 7 | FFN_1 | Feed Forward | 1 × 4096 × 384 |
| 8 | Attention_2 | Multi-Head Attention | 1 × 4096 × 384 |
| 9 | Add_2 | Add | 1 × 4096 × 384 |
| 10 | LayerNorm_2_1 | LayerNorm | 1 × 4096 × 384 |
| 11 | FFN_2 | Feed Forward | 1 × 4096 × 384 |
| 12 | Attention_3 | Multi-Head Attention | 1 × 4096 × 384 |
| 13 | Add_3 | Add | 1 × 4096 × 384 |
| 14 | LayerNorm_3_1 | LayerNorm | 1 × 4096 × 384 |
| 15 | FFN_3 | Feed Forward | 1 × 4096 × 384 |
| 16 | Attention_4 | Multi-Head Attention | 1 × 4096 × 384 |
| 17 | Add_4 | Add | 1 × 4096 × 384 |
| 18 | LayerNorm_4_1 | LayerNorm | 1 × 4096 × 384 |
| 19 | FFN_4 | Feed Forward | 1 × 4096 × 384 |
| 20 | Attention_5 | Multi-Head Attention | 1 × 4096 × 384 |
| 21 | Add_5 | Add | 1 × 4096 × 384 |
| 22 | LayerNorm_5_1 | LayerNorm | 1 × 4096 × 384 |
| 23 | FFN_5 | Feed Forward | 1 × 4096 × 384 |
| 24 | Attention_6 | Multi-Head Attention | 1 × 4096 × 384 |
| 25 | Add_6 | Add | 1 × 4096 × 384 |
| 26 | LayerNorm_6_1 | LayerNorm | 1 × 4096 × 384 |
| 27 | FFN_6 | Feed Forward | 1 × 4096 × 384 |
| 28 | Attention_7 | Multi-Head Attention | 1 × 4096 × 384 |
| 29 | Add_7 | Add | 1 × 4096 × 384 |
| 30 | LayerNorm_7_1 | LayerNorm | 1 × 4096 × 384 |
| 31 | FFN_7 | Feed Forward | 1 × 4096 × 384 |
| 32 | Attention_8 | Multi-Head Attention | 1 × 4096 × 384 |
| 33 | Add_8 | Add | 1 × 4096 × 384 |
| 34 | LayerNorm_8_1 | LayerNorm | 1 × 4096 × 384 |
| 35 | FFN_8 | Feed Forward | 1 × 4096 × 384 |
| 36 | Attention_9 | Multi-Head Attention | 1 × 4096 × 384 |
| 37 | Add_9 | Add | 1 × 4096 × 384 |
| 38 | LayerNorm_9_1 | LayerNorm | 1 × 4096 × 384 |
| 39 | FFN_9 | Feed Forward | 1 × 4096 × 384 |
| 40 | Attention_10 | Multi-Head Attention | 1 × 4096 × 384 |
| 41 | Add_10 | Add | 1 × 4096 × 384 |
| 42 | LayerNorm_10_1 | LayerNorm | 1 × 4096 × 384 |
| 43 | FFN_10 | Feed Forward | 1 × 4096 × 384 |
| 44 | Attention_11 | Multi-Head Attention | 1 × 4096 × 384 |
| 45 | Add_11 | Add | 1 × 4096 × 384 |
| 46 | LayerNorm_11_1 | LayerNorm | 1 × 4096 × 384 |
| 47 | FFN_11 | Feed Forward | 1 × 4096 × 384 |
| 48 | Attention_12 | Multi-Head Attention | 1 × 4096 × 384 |
| 49 | Add_12 | Add | 1 × 4096 × 384 |
| 50 | LayerNorm_12_1 | LayerNorm | 1 × 4096 × 384 |
| 51 | FFN_12 | Feed Forward | 1 × 4096 × 384 |
| 52 | Output | Output | 1 × 4096 × 384 |

## Do this to your own model

```
pip install neurarch-trace
neurarch-trace Maple728/TimeMoE-50M --plan --share
```

- Page: https://neurarch.com/m/timemoe-50m.html
- All models: https://neurarch.com/m/index.json
- The checks: https://neurarch.com/r/index.json