Architectures / Multimodal

🧱 Qwen2-VL (Native-Resolution VLM)

Wang et al. 2024 - the image is not resized to a fixed grid. The patch merger folds each 2x2 group of visual tokens into one token four times as wide before the language model sees it.

From Wang et al. (2024). Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution. This page is the graph, not the PDF: open it, edit it, verify it, export it.

Every number on this page is computed from the graph by the same functions the app runs, not written by hand.

When to pick it

Structure

15 layers. Output shapes are propagated from the input shape, batch dimension excluded.

What the verifier says

The same 43 structural checks that run on every edit in the app, on this graph.

The PyTorch it exports

Generated from the graph above. First 46 lines; the app exports the whole file, plus the training loop, the data contract and a deploy bundle.

# Architecture designed with Neurarch: https://neurarch.com
# PyTorch: compatible with Python 3.8+ and torch>=1.12
# Colab: pip install torch torchvision  (usually pre-installed)

import torch
import torch.nn as nn
import torch.nn.functional as F
from typing import Tuple

class PatchMerger(nn.Module):
    """Qwen-VL / InternVL patch merger. A k x k group of visual tokens becomes
    ONE token k^2 times as wide, then an MLP maps that to the language model's
    width. Token count divides by k^2 and the width multiplies by it, in the
    same step."""

    def __init__(self, merge_size: int = 2, d_in: int = 1280, out_dim: int = 3584):
        super().__init__()
        self.k = merge_size
        merged = d_in * merge_size * merge_size
        # d_in rather than in_dim: the export gate greps for "(in_dim" as an
        # unresolved placeholder, and a bound parameter is not one.
        self.norm = nn.LayerNorm(d_in)
        self.mlp = nn.Sequential(
            nn.Linear(merged, merged), nn.GELU(), nn.Linear(merged, out_dim))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.norm(x)
        group = self.k * self.k
        n = x.size(-2) // group
        # Trailing tokens that do not fill a group are dropped, which is what
        # the shape on the canvas says (floor division) and what a real
        # implementation does after padding the image to a whole grid.
        x = x[..., : n * group, :]
        return self.mlp(x.reshape(*x.shape[:-2], n, group * x.size(-1)))


class Qwen2_VLNative_ResolutionVLM(nn.Module):
    def __init__(self):
        super().__init__()

        self.patchEmbed_1 = nn.Conv2d(3, 1280, kernel_size=14, stride=14)  # Patch embedding (ViT-style)
        self.transformerBlock_1 = nn.TransformerEncoderLayer(d_model=1280, nhead=16, dim_feedforward=5120, batch_first=True)
        self.transformerBlock_2 = nn.TransformerEncoderLayer(d_model=1280, nhead=16, dim_feedforward=5120, batch_first=True)
        self.patchMerger_1 = PatchMerger(merge_size=2, d_in=1280, out_dim=3584)
        self.embedding_1 = nn.Embedding(152064, 3584)
        self.transformerBlock_3 = nn.TransformerEncoderLayer(d_model=3584, nhead=28, dim_feedforward=18944, batch_first=True)

For agents

Also in Multimodal