# Whisper Small vs T5 Small

Speech-to-text against text-to-text, both encoder-decoder.

**T5 Small has 9.7M more parameters than Whisper Small: 14 layers added, 9 removed, 8 changed.**

Source: https://neurarch.com/diff/whisper-small-vs-t5-small.html

## Sides

| | Whisper Small | T5 Small |
|---|---|---|
| Layers | 15 | 20 |
| Parameters | 47M | 57M |
| Input | 80 × 3000 | 1 × 512 |
| Output | 1 × 448 × 51865 | 1 × 128 × 32128 |
| Forward-passes | yes | yes |
| Est. train cost | $0.705 | $0.207 |
| T4 16GB | fits | fits |
| A100 40GB | fits | fits |
| H100 80GB | fits | fits |

## Deltas (T5 Small relative to Whisper Small)

- Parameters: +9.7M (+20.7%)
- Layers: +5
- Added 14, removed 9, changed 8, unchanged 1

## Layer by layer

| # | Status | Whisper Small | Params | Output | T5 Small | Params | Output |
|---|---|---|---|---|---|---|---|
| 1 | changed (shape) | mel_features (Input) |  | 80 × 3000 | encoder_ids (Input) |  | 1 × 512 |
| 2 | changed (shape) | decoder_tokens (Input) |  | 1 × 448 | decoder_ids (Input) |  | 1 × 128 |
| 3 | removed | conv1 (Audio Conv) | 1.5K | 384 × 3000 |  | |  |
| 4 | changed (numEmbeddings, embeddingDim) | token_embed (Embedding) | 20M | 1 × 448 × 384 | shared_embed (Embedding) | 16M | 1 × 512 × 512 |
| 5 | removed | gelu_1 (Gelu) |  | 384 × 3000 |  | |  |
| 6 | removed | dec_pos_emb (Positional Encoding) |  | 1 × 448 × 384 |  | |  |
| 7 | removed | conv2 (Audio Conv) | 1.5K | 384 × 1500 |  | |  |
| 8 | removed | gelu_2 (Gelu) |  | 384 × 1500 |  | |  |
| 9 | removed | to_tokens (Permute) |  | 1500 × 384 |  | |  |
| 10 | removed | enc_pos_emb (Positional Encoding) |  | 1500 × 384 |  | |  |
| 11 | removed | enc_block_1 (Transformer Block) | 1.8M | 1500 × 384 |  | |  |
| 12 | added |  | |  | dec_embed (Embedding) | 16M | 1 × 128 × 512 |
| 13 | added |  | |  | enc_norm (Rms Norm) | 512 | 1 × 512 × 512 |
| 14 | added |  | |  | dec_sa_norm (Rms Norm) | 512 | 1 × 128 × 512 |
| 15 | changed (type, embedDim, numHeads, ffDim) | enc_block_2 (Transformer Block) | 1.8M | 1500 × 384 | enc_self_attn (Multi Head Attention) | 1.1M | 1 × 512 × 512 |
| 16 | added |  | |  | dec_self_attn (Causal Attention) | 1.0M | 1 × 128 × 512 |
| 17 | added |  | |  | enc_residual (Add) |  | 1 × 512 × 512 |
| 18 | added |  | |  | dec_sa_residual (Add) |  | 1 × 128 × 512 |
| 19 | added |  | |  | enc_ffn_norm (Rms Norm) | 512 | 1 × 512 × 512 |
| 20 | added |  | |  | dec_ca_norm (Rms Norm) | 512 | 1 × 128 × 512 |
| 21 | added |  | |  | enc_ffn (Feed Forward) | 2.1M | 1 × 512 × 512 |
| 22 | added |  | |  | enc_ffn_residual (Add) |  | 1 × 512 × 512 |
| 23 | changed (normalizedShape) | enc_norm (Layer Norm) | 768 | 1500 × 384 | enc_out_norm (Layer Norm) | 1.0K | 1 × 512 × 512 |
| 24 | removed | dec_block_1 (Transformer Block) | 1.8M | 1 × 448 × 384 |  | |  |
| 25 | changed (type, embedDim, numHeads, ffDim) | dec_block_2 (Transformer Block) | 1.8M | 1 × 448 × 384 | cross_attn (Multi Head Attention) | 1.1M | 1 × 128 × 512 |
| 26 | added |  | |  | dec_ca_residual (Add) |  | 1 × 128 × 512 |
| 27 | added |  | |  | dec_ffn_norm (Rms Norm) | 512 | 1 × 128 × 512 |
| 28 | added |  | |  | dec_ffn (Feed Forward) | 2.1M | 1 × 128 × 512 |
| 29 | added |  | |  | dec_ffn_residual (Add) |  | 1 × 128 × 512 |
| 30 | changed (normalizedShape) | dec_norm (Layer Norm) | 768 | 1 × 448 × 384 | dec_out_norm (Layer Norm) | 1.0K | 1 × 128 × 512 |
| 31 | changed (outFeatures) | lm_head (Linear) |  | 1 × 448 × 51865 | lm_head (Linear) |  | 1 × 128 × 32128 |
| 32 | same | token_logits (Output) |  | 1 × 448 × 51865 | logits (Output) |  | 1 × 128 × 32128 |

## What this is not

- The two are priced at different declared inputs (80 × 3000 against 1 × 512), so memory, cost and GPU fit are each right about their own model and are not a comparison between them. The layer and parameter deltas are unaffected.
- Parameter counts are derived from the graph, not read from a checkpoint. They are exact for a graph that is fully specified and approximate for one that is not.
- Cost and GPU fit are estimates from the graph under one set of assumptions, not measurements of a run.

## Graphs

- Whisper Small: https://neurarch.com/templates/whisper-small/model.json
- T5 Small: https://neurarch.com/templates/t5-small/model.json
- Check a graph of your own: `POST https://www.neurarch.com/api/v1/plan`