# Qwen2-VL (Native-Resolution VLM)

> Wang et al. 2024 - the image is not resized to a fixed grid. The patch merger folds each 2x2 group of visual tokens into one token four times as wide before the language model sees it.

Pick when input images vary in size or aspect ratio and resizing throws away what matters (documents, charts, screenshots). Watch the token count after the merger: that is the VLM context budget.

- Category: Multimodal
- Paper: Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution (https://arxiv.org/abs/2409.12191)
- Layers: 15
- Parameters: 1.55B
- Input shape (batchless): 3 × 448 × 448
- Output shape: 1 × 320 × 152064
- Verifier verdict: pass
- Graph JSON: https://neurarch.com/templates/qwen2-vl/model.json
- Open on the canvas: https://neurarch.com/?template=qwen2-vl

## Structure

| # | Layer | Type | Parameters | Output shape |
| --- | --- | --- | --- | --- |
| 1 | Image (native resolution) | Input | shape=[3, 448, 448] | 3 × 448 × 448 |
| 2 | Patchify 14x14 | Patch Embed | embedDim=1280, patchSize=14 | 1024 × 1280 |
| 3 | 2D RoPE | RoPE |  | 1024 × 1280 |
| 4 | ViT Block 1 | Transformer Block | embedDim=1280, numHeads=16, ffDim=5120 | 1024 × 1280 |
| 5 | ViT Block 2 | Transformer Block | embedDim=1280, numHeads=16, ffDim=5120 | 1024 × 1280 |
| 6 | Patch Merger 2x2 | Patch Merger (Qwen-VL / InternVL) |  | 256 × 3584 |
| 7 | Text tokens | Input | shape=[1, 64] | 1 × 64 |
| 8 | Token Embed | Embedding | vocabSize=152064 | 1 × 64 × 3584 |
| 9 | to [1, 256, 3584] | Reshape | shape=[1, 256, 3584] | 1 × 256 × 3584 |
| 10 | [image tokens; text tokens] | Concatenate |  | 1 × 320 × 3584 |
| 11 | LLM Block 1 | Transformer Block | embedDim=3584, numHeads=28, ffDim=18944 | 1 × 320 × 3584 |
| 12 | LLM Block 2 | Transformer Block | embedDim=3584, numHeads=28, ffDim=18944 | 1 × 320 × 3584 |
| 13 | RMSNorm | RMSNorm | normalizedShape=3584 | 1 × 320 × 3584 |
| 14 | LM Head | LM Head | hiddenSize=3584, vocabSize=152064 | 1 × 320 × 152064 |
| 15 | next token logits | Output |  | 1 × 320 × 152064 |

## Verifier findings

No finding. Shapes propagate end to end and no advisory rule fires.

## Exported PyTorch (first 46 lines)

```python
# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class PatchMerger(nn.Module):
    """Qwen-VL / InternVL patch merger. A k x k group of visual tokens becomes
    ONE token k^2 times as wide, then an MLP maps that to the language model's
    width. Token count divides by k^2 and the width multiplies by it, in the
    same step."""

    def __init__(self, merge_size: int = 2, d_in: int = 1280, out_dim: int = 3584):
        super().__init__()
        self.k = merge_size
        merged = d_in * merge_size * merge_size
        # d_in rather than in_dim: the export gate greps for "(in_dim" as an
        # unresolved placeholder, and a bound parameter is not one.
        self.norm = nn.LayerNorm(d_in)
        self.mlp = nn.Sequential(
            nn.Linear(merged, merged), nn.GELU(), nn.Linear(merged, out_dim))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.norm(x)
        group = self.k * self.k
        n = x.size(-2) // group
        # Trailing tokens that do not fill a group are dropped, which is what
        # the shape on the canvas says (floor division) and what a real
        # implementation does after padding the image to a whole grid.
        x = x[..., : n * group, :]
        return self.mlp(x.reshape(*x.shape[:-2], n, group * x.size(-1)))


class Qwen2_VLNative_ResolutionVLM(nn.Module):
    def __init__(self):
        super().__init__()

        self.patchEmbed_1 = nn.Conv2d(3, 1280, kernel_size=14, stride=14)  # Patch embedding (ViT-style)
        self.transformerBlock_1 = nn.TransformerEncoderLayer(d_model=1280, nhead=16, dim_feedforward=5120, batch_first=True)
        self.transformerBlock_2 = nn.TransformerEncoderLayer(d_model=1280, nhead=16, dim_feedforward=5120, batch_first=True)
        self.patchMerger_1 = PatchMerger(merge_size=2, d_in=1280, out_dim=3584)
        self.embedding_1 = nn.Embedding(152064, 3584)
        self.transformerBlock_3 = nn.TransformerEncoderLayer(d_model=3584, nhead=28, dim_feedforward=18944, batch_first=True)
```

## Machine access

- Every architecture: https://neurarch.com/a/index.json
- Verify a graph of your own: `POST https://www.neurarch.com/api/v1/check` (see https://neurarch.com/developer.html)
- MCP server, so an agent edits the graph with the checks in the loop: https://neurarch.com/docs/mcp.md