The dependency scanner for open-weight models.
Finds every model your code and deploy configs pull, shows what changed upstream since the commit you approved, flags what is a security risk, and blocks the upgrade in CI.
~/your-repo
$ npx neurarch scan .
1 file read, 1 model reference, 1 high , 3 medium
nomic-ai/nomic-embed-text-v1.5 @ b0753ae76394 embed.py:3
high security auto_map runs code from nomic-ai/nomic-bert-2048; revision= does
not pin it, so that repository's main is what loads
fix: code_revision=<commit> (scan --fix writes it)
medium behavior context window 8192 to 2048 tokens (2026-04-07)
medium behavior declared dtype float32 to bfloat16 (2024-08-26)
Real output of neurarch 0.4.0 on a one-file repository, 2026-11-18; long lines shortened. Each finding links the Hub commit it came from.
Xin Gao · Founder · www.neurarch.com · November 2026
The problem · one model, read from the Hub
You pinned the model. The code it runs is not pinned.
Your code
SentenceTransformer(
"nomic-ai/nomic-embed-text-v1.5",
revision="b0753ae",
trust_remote_code=True)
Model repository
nomic-embed-text-v1.5
@ b0753ae · 2024-06-05
weights, config, tokenizer: pinned
auto_map
Code repository
nomic-bert-2048
@ main · 2025-04-29
revision= does not reach here
nomic-bert-2048: every commit that changed its Python
24 before your pin 23 after it : the code that runs today
Jan 2024
Jan 2025
your pin
2024-06-05
Jun 2024
main today
2024-01-04: 5 commits changing Python 2024-01-31: 3 commits changing Python 2024-02-23: 3 commits changing Python 2024-03-19: 3 commits changing Python 2024-03-29: 1 commit changing Python 2024-05-02: 6, 2024-05-03: 1 commits changing Python 2024-06-05: 2 commits changing Python 2024-08-26: 3 commits changing Python (after the pin) 2024-10-01: 3 commits changing Python (after the pin) 2024-11-18: 5, 2024-11-19: 1 commits changing Python (after the pin) 2024-12-18: 2 commits changing Python (after the pin) 2025-02-07: 2, 2025-02-11: 5 commits changing Python (after the pin) 2025-02-24: 1 commit changing Python (after the pin) 2025-04-29: 1 commit changing Python (after the pin)
17 of 150
popular models that run repository code load it from another repository
table ↗
19.3M
downloads of those 17 in the last 30 days
Hub API, 2026-11-17
11 of 17
had that code change since they started pointing at it (18.3M downloads)
commits ↗
The problem, in public code · 353 files read by hand
The teams that pin are the ones it catches.
353 public Python files load one of the 17
one square per file; GitHub code search, every file read
17 pin a commit with revision=
336 do not pin at all
167 load through sentence-transformers · 5 mention code_revision
A maintainer in 2024 (#2613 ↗ ), on fixing it:
"very few models use third party repositories"
On popular models: 17 of 150, 19.3M downloads a month.
MTEB, the embedding leaderboard
Pins nomic-embed-text-v1.5 at b0753ae.
The code that runs is nomic-bert-2048 at
its 2025-04-29 head: 23 Python changes later ,
including "fix: dtype" and "fix: mask".
mteb/.../nomic_models.py ↗
The same model, pinned at its first commit
e178409 (2024-02-10) points at a code repo
whose main deleted the file. Loading it today:
OSError: ... does not appear to
have a file named configuration_
hf_nomic_bert.py
GitHub code search for each of the 17 ids with trust_remote_code, every result read, 2026-11-17. Maintainer quote: sentence-transformers #2613 . MTEB pin: nomic_models.py . Both reproduced with AutoConfig, no GPU. MTEB has not been contacted yet; a courtesy issue goes to them before anything is posted.
The problem, counted · 80 public incidents and the change tracker
Publishers change model files often, and the config shows the cause in 26 of 30 incidents.
Where the change came from, and whether the config shows the cause
The model's publisher changed a file (30)
template, EOS, tokenizer, config, a new revision
26 readable from the config
4 partly, 0 not
An engine, framework or provider changed (50)
version upgrades, kernels, quantization loaders, flags
13
19 partly
18 not: kernel and implementation bugs
How often the publisher changes it, after release
114 of 435 models with a chat template changed it later,
and the rendered prompt changed (26.2%, 2026-11-17); 95 within 30 days.
84 of 672 changed the EOS value (12.5%).
First public report: median 4 days after the change; the tail runs 76 to 232 days.
Three that shipped without an error
gpt-oss-120b · one line, 76 days
<|call|> added to the stop tokens. Parallel tool
calls collapsed to one; first reported 76 days later.
HF openai/gpt-oss-120b discussion #151 ↗
DeepSeek-R1 · 9 bytes, weights unchanged
The template began prefilling <think>. Day 4:
reasoning output went null in vLLM.
HF deepseek-ai/DeepSeek-R1 discussions #144, #181 ↗
Qwen2.5-7B-Instruct-GGUF · still stale
The official GGUF ships the tool template the base
fixed on 2024-09-19; 181,341 downloads in 30 days.
Read from the GGUF header, 2026-10-19
80 open-weight incidents found by searching eight engine and framework repositories and the discussion boards of 50 popular models, each read in full; two closed-model postmortems excluded. Public reports only, so silent regressions are missing; "readable from the config" is a judgement written down per incident. Template and EOS shares: the change tracker's 672 snapshotted models (435 with a chat template), history read to 2026-11-17; "within 30 days" is of the first recorded commit. Report delay is to the first public report, not to when a team noticed.
The problem, security side · public advisories, dated
Model files are an attack surface: 9 public advisories and reports since 2024, 6 of them in 2026.
2024
2025
2026
May 2024 · critical llama-cpp-python: a GGUF chat template runs code CVE-2024-34359 ↗
Apr 2025 · critical PyTorch: weights_only=True still runs a crafted file CVE-2025-32434 ↗
Jan 2026 · high vLLM ran auto_map code without checking trust_remote_code CVE-2026-22807 ↗
Apr 2026 · critical · CVE-2026-5760 ↗ SGLang: a model's template runs code
Jun 2026 · medium vLLM: pinned deployments load unpinned code CVE-2026-47155 ↗
Sep 2025 · Unit 42 ↗ deleted author re-registered, model replaced under the same id; fix: pin a commit
Feb 2026 · ICLR template-only backdoor: 18 models, 4 engines arXiv 2602.04653 ↗
May 2026 · high · CVE-2026-4372 ↗ transformers: one config field loads another repo's code
Oct 2026 ↗ vLLM: a request's code_revision runs code
What scan reads in our 608 tracked models: 150 run repository code (17 from another repo) · 166 template changes altered the prompt, each rendered at both commits · 15 ship pickle weights only.
Dates are GitHub advisory publication dates (34359 , 32434 , 22807 , 5760 , 4372 , 47155 , GHSA-h3rc ), read 2026-11-18. Counts: Hub API, 2026-11-17. Most recorded changes are behavior, not security; the scan reports the two apart and never calls a template edit a vulnerability.
Why now
Open-weight models change more often, and more of each change lands before the tools support it.
New Hugging Face uploads whose
architecture transformers, vLLM
or llama.cpp did not support
that day
8.1%
2024 Q2
16.0%
2026 Q3
HF Hub, Neurarch analysis
2026-10-18; ~35,000 uploads a
quarter; class-name match
"Wrong output" issues naming
a model, 9 inference and
tooling repositories, per month
~12
2024 Q1
~75
2026 Q3
GitHub issues, Neurarch analysis
2026-10-18; title keywords; 60%
to 72% true on a sample of 40
Pull requests signed by a
coding agent, 931 PyTorch-
ecosystem ML repositories
1.5%
Aug 2025
16.0%
Sep 2026
GitHub, Neurarch analysis
2026-10-17; 319,799 pull
requests since 2023
Open-weight models' share of
tokens on OpenRouter, one week
72%
100%
week to 2026-09-22
OpenRouter, "The token state
of play"; closed models still
take 73% of spend
Every change here is a moment someone has to decide whether it is safe to run or serve. The growth curves are the market's, not ours. Upload-day support is matched by class name, so a renamed class counts as unsupported.
The product · one command, in CI
Every model your code pulls, what changed since your pin, and the fix as a diff.
1 · Find every reference
Python calls, and ids held in variables
registry records (MTEB)
k8s, compose, Dockerfile env
vllm / sglang launch flags
MTEB repo: 2 found before records were read, 923 after
2 · Your pin vs today
config, tokenizer, template
auto_map and the repos it pulls code from
weight format
the daily change record
no weights downloaded, no GPU, no model run
3 · Three lanes, never mixed
security code on load, pickle, CVE fields
behavior template, tokenizer, defaults
hygiene unpinned, unreadable, renamed
4 · Out, where it is read
exit code that blocks the PR
--fix : pins written into code
--cyclonedx : ML-BOM, mappedto NIST SP 800-218A
OSV advisory feed
Renovate PR when a pin moves
before embed.py
model = SentenceTransformer(
"nomic-ai/nomic-embed-text-v1.5",
revision="b0753ae76394dd36bcfb...",
trust_remote_code=True,
)
after neurarch scan --fix
trust_remote_code=True,
+ model_kwargs={"code_revision": "7710840340a0..."},
+ config_kwargs={"code_revision": "7710840340a0..."},
)
re-scan: 0 high. Nothing you run changes today;
it just stops changing underneath you.
Real run, 2026-11-18, neurarch 0.4.0 on npm (--cyclonedx writes its file from 0.4.1; on 0.4.0 use --bom ). The BOM validates against CycloneDX 1.6 in strict mode. MTEB: 916 of the 923 are ModelMeta registry records, spot-checked by hand. Weights and outputs are never read; the scan compares files and code references, not what the model says.
Why believe it · what changed, recorded and published
754 recorded changes to 672 open-weight models, in formats other people's automation can read.
754 changes, by the year of the commit
2019: 4 recorded changes 4 2019 2020: 39 recorded changes 39 2020 2021: 10 recorded changes 10 2021 2022: 19 recorded changes 19 2022 2023: 44 recorded changes 44 2023 2024: 261 recorded changes 261 2024 2025: 214 recorded changes 214 2025 2026: 163 recorded changes 163 2026
Only models tracked today, so early years are undercounted; 2026 runs
to Nov 17. 311 of 672 models changed a tracked file after upload.
By what changed
tokenizer 244 chat template 220 structural config 213 defaults, metadata, format 77
Published as What it does
OSV advisories All 754 validate against the official schema; CC BY 4.0
Renovate datasource Opens a pull request when a pinned model moves
pin-check Unpinned downloads. Held out, 18 files: 0 missed
scan + ML-BOM Three lanes per model; CycloneDX 1.6, strict-valid, mapped to NIST SP 800-218A
Feeds live at neurarch.com/changes-feeds/ ; tracker figures as of 2026-11-17. pin-check and scan are on npm, neurarch 0.4.0 (2026-11-18).
Neurarch model change tracker, first run 2026-10-18: each file's Hugging Face commit history read back to 2019 and diffed key by key. Only models that still exist are seen. Most changes are the publisher's fixes, not bugs; the record says what changed, not who was wrong.
Competition · what each one reads, from its own documentation
Scanners judge a file at one moment. None we checked compares your pin with today.
CVE in vllm, torch
malicious payload in a file
finds model ids in code + deploy
diff since your pin
code from another repo
blocks in CI, writes the fix
Package scanners Snyk, Socket, Dependabot
Artifact proxies JFrog, Sonatype (proxy the Hub)
Model file scanners Protect AI (Palo Alto), HiddenLayer, HF
Lint rule Bandit B615 (unpinned download)
Neurarch scan reads all three lanes
does it
in part
not in its docs
Neurarch's halves: engine-version CVEs only (vLLM, transformers); pickle format, not payload content
The category already has buyers and exits: Palo Alto Networks bought Protect AI (SecurityWeek ); JFrog scans Hugging Face models with Hugging Face (JFrog ). We sit next to them: they say a file is malicious, we say it changed under your pin.
Earlier stage map (pre-check story): Package and supply-chain scanners: Snyk, Socket, Prisma Cloud · JFrog and Sonatype proxy Hugging Face models; leaves They flag CVEs in vllm or transformers, and store model revisions; the diff of a model repository between two commits, and code it pulls from another repository. Model file scanners: Protect AI (Palo Alto Networks, 2025), HiddenLayer, Hugging Face's own scans; leaves They look for a malicious payload in a file. A template, tokenizer or default that changed under your pin is not malware, and a published template backdoor evaded the Hub's scans (arXiv 2602.04653). Training runtime, stragglers: Clockwork, $73M raised (HPCwire) · NVIDIA NVRx, free; leaves They act during the run. The share of a slowdown that the split decides before launch. Scheduling and admission: Expanse, $5.3M seed (Pulse 2.0) · NVIDIA Slurm, Run:ai, KAI; leaves A new job, recipe or split with no telemetry history yet. Fine-tune, post-training: Unsloth, LLaMA-Factory, TRL, verl · Predibase, bought by Rubrik (CNBC); leaves One check across their config dialects; verl's docs tune rollout memory by trial and OOM. Inference sizing: NVIDIA AIConfigurator, the HF hardware filter, free calculators; leaves MoE, MLA and hybrid-SSM state memory; the diff when a model is swapped. Serving parity: Moonshot K2 vendor verifier, OpenAI gpt-oss verification: free, one model family each; leaves Before deploy, across families. Output parity stays theirs. New-model bring-up: Chip-vendor teams · Inferact, $150M seed (Yahoo Finance) · Red Hat; leaves The structural diff that says what is new in a model. Agent governance: Runlayer · sandboxes (Modal, E2B, Daytona); leaves The GPU job itself. We are a policy they call, not a rival. Marks are what each product's public documentation says it reads, as we read it in November 2026; a product may do more than its docs say. Proxies see a model id only when the download goes through them. Bandit B615 flags an unpinned from_pretrained call, which is what pin-check reproduces.
Who buys it · and the evidence they already want it
AppSec pays, the ML platform team runs it, and NIST already asks for exactly this check.
ML engineer adds or bumps a model in a pull request
neurarch scan in CI blocks on security, comments on behavior, writes the BOM
ML platform team uses it: owns the model list and the pins
AppSec / ProdSec pays: owns the scanner budget and the policy
What an auditor asks for, by name
SP 800-218A PW.4.4 verify acquired AI models,
including configuration, before use (High priority)
PS.3.1, AI 600-1 GV-1.6-003 track model versions
each scan finding carries the item it maps to
NIST SP 800-218A is a recommendation, not a certification, and the BOM says so. Threads: Dynamo #11369 , exo #2085 , GHSA-3ww4 ; none of these projects is a customer or has been contacted. The buyer split is our hypothesis, being tested in discovery calls now.
What we do not claim
Most changes are fixes, not attacks. The scan says which lane a change is in and never inflates it.
222 template edits, each rendered at both commits
166 changed the prompt
56 cosmetic
0 trigger-shaped (security)
We once counted 9; all were Qwen2.5-VL testing a content field. Corrected in
public (#528). 18 more edits could not be rendered and are labelled so.
17 code repositories that revision= does not pin
0 with malicious code found. The risk is that the pin does not
decide what runs, not that anyone has abused it yet.
A first read is a baseline, never an alert.
It never says
this model is malicious this template edit is a vulnerability this output is correct we save you X
It only says
which models you pull, and where what changed since your pin, with the commit which of those runs code or can execute the one-line fix
An incumbent adding "model changed" as a checkbox has to sort lanes the same way or it pages AppSec for every tokenizer fix, which is how a scanner gets turned off. Behavior changes still matter: a 9-byte DeepSeek-R1 template change nulled reasoning output in vLLM (slide 4).
Where it is still weak
Three weaknesses, each with a number.
0
outside teams running scan (on npm since 2026-11-18)
outside users: 0 8 US teams to call
Next: 8 discovery calls; MTEB notified first.
1 hub
Hugging Face only; other registries not seen
Hugging Face others: not yet 608 models tracked daily
Next: OSV ecosystem proposal for Hub models, drafted.
$0
revenue. No paying customer yet
paid: 0 first price: the $3,000 report
Next: the report runs the scan on their repos.
Also still true: the 17-of-150 count is on our list of popular models, not a census of the Hub; scan reads Python, notebooks are not edited by --fix ; and the engine-sizing checks underneath (backup) have their own open measurements.
Traction
Small, and here is exactly how small. No revenue, and no call yet from a key that is not ours.
17
outside signups, of 24 accounts
no revenue · Supabase, 2026-10-18
1,496
npm downloads of the MCP server, last 30 days
npm ↗ half on 3 release days
291
npm downloads of the neurarch CLI, last 30 days
298
machine calls to the check; 0 from a key that is not ours
295 unattributable · 2026-10-18
2%
14-day retention of the free visitor tool
12 of 679 matured first visits
neurarch-mcp · downloads per day, 10-19 to 11-17
neurarch CLI · downloads per day, same 30 days
10-21: 79 downloads 10-22: 14 downloads 10-23: 355 downloads (release day) 355 10-24: 30 downloads 10-25: 34 downloads 10-26: 68 downloads 10-27: 24 downloads 10-28: 63 downloads 10-29: 40 downloads 10-30: 23 downloads 10-31: 17 downloads 11-01: 26 downloads 11-02: 32 downloads 11-03: 12 downloads 11-04: 162 downloads (release day) 11-05: 232 downloads (release day) 232 11-06: 23 downloads 11-07: 40 downloads 11-08: 25 downloads 11-09: 27 downloads 11-10: 21 downloads 11-11: 21 downloads 11-12: 43 downloads 11-13: 38 downloads 11-14: 12 downloads 11-15: 12 downloads 11-16: 11 downloads 11-17: 12 downloads
10-21: 32 downloads 10-22: 12 downloads 10-24: 6 downloads 10-25: 24 downloads 10-26: 4 downloads 10-27: 39 downloads 39 10-28: 8 downloads 10-29: 9 downloads 10-30: 10 downloads 10-31: 7 downloads 11-01: 7 downloads 11-02: 15 downloads 11-03: 7 downloads 11-04: 2 downloads 11-05: 5 downloads 11-06: 10 downloads 11-07: 3 downloads 11-08: 15 downloads 11-09: 3 downloads 11-10: 6 downloads 11-11: 8 downloads 11-12: 3 downloads 11-13: 5 downloads 11-14: 8 downloads 11-15: 14 downloads 11-16: 8 downloads 11-17: 21 downloads
release day: mirrors fetch every new version (749 of the 1,496)
the launch burst, 899 on 10-18, is outside this window on purpose
A tool a person must reopen
2% came back. The rows it produced are unattributable.
We stopped building for this user in September.
A scan that runs in CI on every pull request
Nobody has to come back: it runs where the model is added.
30-day test, decided by a payment on 2026-11-16.
Who pays, and how
Entry is a $3,000 report. The subscription is per managed GPU, free under 8.
First buyer
The platform or inference lead who self-hosts open weights on 64 or more GPUs, and both fine-tunes and serves. Every base-model swap, recipe, split and serving config is theirs to approve.
Second segment, to validate
Chip-vendor model bring-up, inference-provider onboarding, and model labs before release. They already pay for day-0 support (evidence in notes).
Free
under 8 GPUs; the CLI and the MCP server are free
$3,000 report, 30 days, flat
requested vs actual over their jobs, plus a pre-check
of every change they made; credited to a subscription
$15 per managed GPU a month
billed annually ($180 a GPU-year), no minimum;
Enterprise $25 adds SSO and supported self-host
Comparable: Run:ai lists $2,600 per GPU a year (AWS Marketplace).
It schedules the GPUs; we decide before a change takes one.
NVIDIA posts a manager role whose job is making day-0 model launches repeatable, base $224,000 to $431,250 (job posting, Built In LA) · AMD and Google work with Inferact on day-0 support (Inferact announcements). Price for this segment: set by the 30-day test. Prices are ours, decided 2026-10-18, minimum removed 2026-10-19; none has been paid yet. A managed GPU is one that passed through the admission hook that month; blocking is never the paywall. Teams that use only pull requests may pay $20 per active committer a month instead. Run:ai price from its AWS Marketplace listing.
Market, bottom up · at our list price per GPU
At $180 per GPU a year: $0.8B this year, doubling by 2030 with the fleets.
Fleet size
Organizations
GPU spend / yr
GPUs x $180
8 to 63 GPUs
~14,000
$0.36M
$5.8K
64 to 499
~4,700
$2.7M
$40.5K
500 to 4,999
~1,100
$22M
$281K
5,000+
~140
$300M+
$2.25M
Violet edge: where we sell first, the wedge (64+ GPUs);
without the 5,000+ that build their own, $0.50B.
$0.81B
2026
$1.6B
2030
Wedge
64+ GPU fleets
$1.1B
2026
$2.1B
2030
Serviceable
+ small teams, inference
$70M to $350M
2026
$135M to $675M
2030
GPU-cloud channel
$0.01 to $0.05 per GPU-hour
2026 2030 faded = high end of a range
Organization counts are modelled (installed base of ~17M H100-equivalents x band share / typical fleet), not a census; Epoch's cluster database gives floors of 50 to 84 per band in the wedge. Priced at our list, $180 per GPU-year: each band's typical fleet (30, 225, about 1,560 and 12,500 GPUs) x $180. Comparable only: Run:ai, $2,600 per GPU-year on AWS Marketplace. The GPU-cloud channel is priced per GPU-hour and is unchanged. Rented GPU-hours, 7.0B in 2026. Fleets are counted by size, not by whether they run open weights: counted only on GPUs that run open-weight models on clusters their owners schedule, the serviceable figure is about $170M a year ($30M to $580M; Epoch AI chip census x ETR enterprise survey). The bring-up segment is not in these bars.
What thickens with time · measured, not asserted
Anyone can rebuild the change list from git. Not what we watched happen.
Rebuildable from git history today
749 of 754 change rows (99%): not a moat, and we say so
Only exists because we were watching
576 hashed weight snapshots; 7 weight changes caught
9.9 h median from commit to our record
8,980 model-days of download counts, 15 days so far
2 commits later erased from main; git no longer has them
measure:reconstructible, 2026-11-17; both erased heads in one GGUF repo.
And three that need someone else
Read by other people's tools
OSV feed, Renovate, ML-BOM, badges
today: 0 outside readers
The customer's own record
every pin, finding and waiver, per repo
today: none; starts with the first customer
The position in CI
a required check; removing it is a policy change
today: none; starts with the first install
The risk, said first: Hugging Face shows code-repo commits on the model page, or a package scanner adds model ids. The history before that day stays ours; the race is to be the check teams already run.
Team · the next 30 days · the ask
One person built all of it. Next: a payment, an 8-GPU measurement, two buyer replies.
Xin Gao, founder
Eight years in industry: machine learning engineer, then lead applied scientist and AI research scientist. PhD. Solo founder; built every box on the product slide.
The ask A pre-seed round is open; terms in the conversation. Use of funds, each tied to a number on these slides: the first 3 paying organisations, layout and memory calibration on 8 GPUs and more, and a bring-up triage pilot.
1 · Pay one paid $3,000 change pre-check report, from 20 self-hosting teams
2 · Layout on 8 x H100 the recommended split is faster, by at least half the prediction
3 · Bring-up a chip vendor or inference provider asks for a second triage report
4 · Config watch 3 teams say an alert would change a decision; 1 connects its model list
5 · Outside calls 5 keys that are not ours call the check (today: 0)
The 2026-11-16 decision, written down now
1 closes raise on the change pre-check; 2 to 4 are the evidence
only 3 signals first buyer becomes chip vendors and providers; 30 more days
1, 3 and 4 fail we say so, and change what we sell
www.neurarch.com · the bot, the MCP server, the CLI, the calibration table, the model pages and every dataset are public today. If test 2 fails, the deck drops "balanced" and keeps legality and per-stage memory.
Backup
Backup: the engine under the scan.
These slides are the earlier pre-check story. Every number in them still stands; they are here as proof of the depth behind the scan, not as the pitch.
The problem
A change breaks in places that raise no error. Most of it is readable from the config before launch.
of LLM training jobs: the last pipeline stage (LM head and loss)
caused over half of the slowdown
ByteDance, OSDI 2025 (arXiv 2505.05713)
39.3%
of jobs slowed by uneven sequence lengths, 1.34x on average
ByteDance, OSDI 2025; slowed = fwd/bwd time correlation ≥ 0.9
21.4%
of 929 inference-engine bugs had a configuration root cause
Liu et al., arXiv 2506.09713 (TOSEM)
23%
of the 406 most-downloaded HF models with a config changed a
structural key after release (vocab, max positions, hidden size, rope)
HF Hub config history, Neurarch analysis 2026-10-17
23%
Worker problems, hardware or software, caused over half the slowdown
in only 1.7% of straggling jobs (ByteDance, OSDI 2025).
The rest was the job's own shape.
Same weights, different config
gpt-oss-120b, 20 agent tasks, same requests
Ollama
0.650
vllm-mlx
0.400
cause: the chat template drifts from the
training format in 3 places. vllm-mlx #568
Kimi K2 tool-call schema accuracy
official
100%
vLLM
76.00%
Together
71.96%
Moonshot K2 vendor verifier (K2-0905)
Nothing crashed. A standard eval does not
see it until someone runs the right one.
When it is found today
Today a change is found out after it holds a GPU or serves traffic. The answers exist at submit.
The change What goes wrong Found today The pre-check answers at submit
Swap a model or checkpoint A structural key changed after release: vocab, max positions, rope, hidden size at load, or in eval What changed: structure and config diff, before anything downloads weights
New fine-tune or RL recipe Peak memory over the card; data longer than the config accepts first batch, OOM What it needs: training peak per card, smallest card, cost, data fit
New TP / PP / DP layout An illegal split; the last stage carries the head and loss; uneven sequence lengths in a trace, after the run Is the layout sound: legality, per-stage work and memory, bubble, sequence imbalance
New serving config Context past what rope was trained for; TP against KV heads; template or sampling drift in user reports Does it match the reference: the configuration, not the outputs
Terraform plans the infrastructure change before it is applied. CodeQL scans the code change before it merges. A model change still waits for the GPU or for the users. "Found today" follows the public record on the previous two slides: the straggler study diagnosed from traces after the run; the template and vendor-verifier gaps were found by running evals after deploy. Anthropic's September 2025 postmortem: two of three issues were configuration or deployment changes, and evals did not catch the degradation users reported.
Demand, in public · GitHub threads, November 2025 to October 2026
Teams write parts of this check by hand, one project at a time.
Answer Thread, opened What they built or asked for
What it needs vLLM #29325 , 2025-11-24A TP and max-length planner without installing vLLM; closed, not planned
What it needs Unsloth #5225 , 2026-04-30Patches its own KV estimate for a new attention pattern
What it needs Ollama #17099 , 2026-07-09A new release inflated the estimate; the model no longer fit
What it needs FastVideo #1978 , 2026-10-18Adds a VRAM calculator to its deployment cookbook
Layout vLLM #50576 , 2026-07-31Last pipeline rank carries the LM head; layers re-split by hand
Layout vLLM #52435 , 2026-08-15Experts not divisible by the EP size: every worker crashes
What changed exo #2085 , 2026-05-11Upstream repo dropped a tokenizer file; the model stopped loading
What changed Dynamo #11369 , 2026-07-07Pins Hugging Face resolution to the worker's commit SHA
Reference LM Studio #2012 , 2026-06-05Chat template calls an undefined macro; tool calls crash
Public issues and pull requests, all read on 2026-10-18; the date is when each was opened. Cited as evidence that the problem exists and is solved piecemeal; none of these projects is a customer or has been contacted.
The product
One typed graph of the change. Four answers, enforced where the job is admitted.
Typed structural graph
from config.json, or a trace
195 layer types · 50 structural checks
no weights · no GPU · deterministic
the same answer to a person,
an agent, CI
What changed Config, tokenizer, template, licence: per revision, 472 models. live
What it needs Training and serving peak per card, smallest card, cost, data fit. live
Is the layout sound TP/PP/DP/CP/EP legality, per-stage work and memory, bubble. live
Does it match the reference A serving launch: context, TP, KV memory, template, sampling. live
Where it is enforced
SLURM cli_filter · built
SkyPilot policy · built
Claude Code hook · live
MCP · live
API · live
PR bot · live
offline · built
live: running in public today (npm, the API, a public repository) · built: merged and tested, release next · building: not shipped. The CLI was on npm at 0.3.0 when this slide was written (0.4.0 since 2026-11-18) with the layout check, serve-check, data fit and pin-check; neurarch-mcp 0.21.6 carries the first two as check_parallel_layout and check_serving_config. The change record and its feeds are live at neurarch.com/changes/ ; scan, template-check and the newest serve-check work are merged and ship in the next CLI release. Generation is free and not the product; the answers and their record are.
What a caller gets back · real output, two commands
A last stage 24% slower than the mean, and a context change that no longer fits.
what it needs seq_len: 1024 → 2048
DocLM ⛔ no longer fits A10G 24 GB
Peak training memory 38.4 GB at batch 32 x 2,048
needs 44.2 GB with the 1.15x headroom
vs base +17.8 GB (20.6 GB at 1,024)
largest batch that fits: 16 , 2 accumulation steps
smallest card that holds it: L40S 48 GB
commit 2, batch 16 x 2: fits, 20.6 GB
is the layout sound TP 4 x PP 4 x DP 8
Qwen/Qwen3-8B · 128 x H100 80 GB
stage extra vs mean peak/GPU
0 embedding 0.92x 16.8 GB
1 - 0.92x 12.4 GB
2 - 0.92x 11.1 GB
3 head+loss 1.24x 14.8 GB
split 10,10,10,6 brings the slowest stage to 1.02x
derived from the config, not timed on a GPU
Neither change raised an error in review. Both answers existed before the job was submitted, and both are kept with the change.
Left: neurarch-ai/model-ci-example#11, a public pull request; neurarch-bot traced the model on the runner and edited one comment from red to green. Right: npx neurarch layout Qwen/Qwen3-8B --tp 4 --pp 4 --dp 8 --global-batch 512 --seq-len 4096 --gpu h100 , from the CLI README. Memory is an estimate on the calibration of the next slide; stage time is predicted from FLOPs and memory bandwidth, checked against pipeline runs on 2 and 4 L4s, and it overprices the head on H100.
Why believe it · the layout check and the config read, measured
The last stage was heavier on every GPU run; 68 of 81 parameter counts land within 2%.
LM head + loss against one block: measured / predicted
within 30%
1.4
1.2
1.0
0.8
0.6
1.23
1.11
1.28
0.72
0.99
L4, 3 runs
H100, 2 compute-bound runs
The heavier side was right on all 5. Per-unit times summed to
within 1.4% to 2.5% of the whole step. The size varies by card.
Parameters from config.json, no weights read,
against 81 comparable Hugging Face checkpoints
68 within 2%
13 not
0.3%
median error, published at neurarch.com/m/
Every miss is printed on the same page as the hits.
Split A/B on 2 x H100, PP 2
Recommended split beat the even one on every
repeat; predicted gain 1.5 points off measured.
Not reliable yet
Absolute step time, about 24% high on H100.
We quote gains, never a predicted step time.
Layout: 2026-10-18 on Modal, one L4 and one H100, models built from config with random weights; per-block forward and backward timed, five runs. Direction and size of the imbalance only, not a speedup. A/B: Qwen3-4B on 2 x H100, 1F1B, two repeats per split; the gain error is in-sample (these runs were used to review the cost curve), and PP 4 is not run yet. Parameters: public/m/index.json , 234 model pages; quantized, custom-code and multi-tower checkpoints are excluded from the 81 and listed.
Why believe it · memory: 203 measured runs
Held out, it flagged 113 of the 131 times a run did not fit a card; accelerate, at most 8.
97 runs of 1 GB or more, one dot per run per method
1
4
16
64
≤0.1
1
4
16
64
measured peak on the GPU, GB (log)
predicted, GB (log)
Neurarch, held out by architecture
accelerate estimate-memory, float32
predicted = measured
Of 131 times a run did not fit a card, said so
Neurarch, held out
113
accelerate, float32
8
accelerate, bf16
0
Median error, runs of 1 GB or more
7%
held out by architecture
14%
on 25 runs measured
after the fit
The out-of-sample number is the one to hold us to.
public/memory-calibration.json: 203 runs, 178 in the fit and 25 measured after it; 55 ran out of memory. accelerate cannot see activations, so it says "fits" on runs that crash. Card-verdict agreement is not quoted: accelerate matches us there. Also measured: 96 of 96 blocked graphs crashed in PyTorch, 80 of 80 passed ran, 24/24 verifier-passed designs trained end to end on a managed T4.
Why believe it · the serving check, against what vLLM itself printed
Out of sample: VL KV memory within a median 5.96% of vLLM's log, CUDA graphs within 0.38 GiB.
Vision-language launches, KV memory: median error
10 public vLLM logs, none used to fit
before
9.35%
now
5.96%
the vision tower's profiling pass is now counted
0.38 GiB
CUDA graph memory, mean error against
vLLM's printed estimate, 10 logs held out
was 0.63 GiB on the same 10 logs
1.07%
KV token count error,
out of sample
End to end on public logs, KV memory: median 7.6% off.
The mean is pulled up by a few outliers, so we do
not quote it. Nothing here ran on a GPU: every
prediction is from config files.
What it reads now
Multimodal configs as the language model. TP, KV heads and length come from the text config, not the vision tower listed first.
Weights against the checkpoint. The predicted size is checked against the published one; a mismatch is a warning, and memory is sized on the published bytes.
New cards. A40, GH200 96 GB and 144 GB, RTX PRO 6000, and Ascend 910B for memory only.
A receipt. After launch, what vLLM measured, printed next to what was predicted.
On npm since neurarch 0.4.0 (2026-11-18).
Neurarch serve-check validation, 2026-10-18, against public vLLM startup logs. The CUDA graph figure leaves out the one issue a coefficient was read from. Out of sample means the log was not used to fit anything.
What we do not claim
A legal design is not a good design, and we publish the number that says so.
0.17
Spearman correlation between our static score and the trained outcome, across 24 GPU-trained designs. Essentially uninformative.
Measured, and published when wrong
An agent edited Karpathy's GPT for 3 hours on an H100 with the check in shadow: 30 changes, 0 refused, 0 crashed, parameter count within 0.25% on every row. The GPU-fit line was wrong on 27 of 28 rows; that failure started the memory calibration two slides back.
So the product never says
"This design will train better." · "We save you X% of GPU spend." · "The model got it wrong." · "This deployment produces correct outputs."
It only says
What changed. How much memory at your batch, whether it fits your card, and what one run costs, with the assumptions. Whether a split is legal and which stage is heaviest. Whether the serving configuration matches the reference: the configuration, never the outputs. And it keeps that answer.
SELECTORS.md. A frontier model reading the exported code ranks held-out pairs at 80%; we abstain or score 56.7%. Published, not hidden. The honest boundary is part of the product: an incumbent adding "will it fit" as a checkbox has to publish its misses too, or it is a feature nobody can trust. Output parity is what the model vendors' own verifiers measure, by running both implementations.
Backup · what breaks in training repositories, counted
100,909 issues from 47 training repos: one in six is a failure, half of those at startup.
Share of 15,486 failure reports: whole populations of 43 repos,
keyword samples of 4 giants
Other RuntimeError 24.5%
Out of memory 18.2%
CUDA driver / kernel 13.5%
Shape / size mismatch 12.5%
NCCL / distributed hang 9.6%
Config error (heads, rope) 8.4%
dtype / device 6.8%
NaN / inf 6.5%
Share a pre-launch structural read could catch
5.5% after hand-checking
up to 36% if every memory and shape report is counted
Of 2,112 reports that say when the run died
(regex, unaudited)
53%
at startup
Startup, first step: 53%
Early, under 15 min: 18%
Hours in: 17%
rest: mid-run or vague
Inference and diffusion apps die at startup
70% of the time.
Cheap in GPU time. Expensive in iterations: each
one is a queue slot, a restart, and an agent turn
spent reading a stack trace.
Neurarch study, 2026-10-17: 100,909 human-filed issues from 47 repositories (whole populations where GitHub's listing allows, newest 9,900 otherwise; pytorch, transformers, vllm and ultralytics keyword-sampled), 15,486 with a failure signature, classified by error string. 120 rows read by hand: shape labels 78% right, config labels 33% right; the certain share is corrected for that. Same read at 931 ML repositories (291,701 issues, 19,056 failure reports): out of memory 16.2% of failures, structural share 0.7% of all issues.
Backup · why we do not pitch GPU savings
In well-run labs, the failures a static check catches are the fast ones.
Failed GPU jobs, Acme cluster (Shanghai AI Lab, NSDI'24), 152,838 jobs
Share of failed jobs that die within 10 minutes
96.4%
Share of failed GPU-time those fast failures consume
5.3%
Share of failed GPU-time attributed to infrastructure (NVLink, ECC, network), not the model
82%
Hu et al., NSDI 2024, Table 3 (infrastructure share); the two bars above re-computed by us from the public AcmeTrace on 2026-10-16: 152,838 failed GPU jobs on Seren, median 0.4 min to failure. Data: Shanghai AI Lab Acme trace (InternLM/AcmeTrace), CC BY 4.0; Hu et al., Characterization of Large Language Model Development in the Datacenter, NSDI '24. These are experienced labs; no public trace exists for teams whose agents launch runs, which is why we are recording one.
Backup · the same failure classes, by source and by repository type
On whole issue populations, the failure tail moves by repository type.
GitHub issues by repository type, share of failure issues
core fw orchestr. fine-tune model libs inference
CUDA OOM
7.6
9.2
11.3
6.2
12.4
shape
11.5
8.6
13.0
21.1
6.9
NCCL
12.8
21.6
17.2
3.8
7.1
config
7.6
12.2
10.7
8.4
11.3
lower bound
15.1 14.9 14.1 13.1 11.9
core fw: pytorch, Lightning, torchtitan, fairseq · orchestration:
accelerate, DeepSpeed, Megatron, ColossalAI + 3 · fine-tune: peft,
trl, unsloth, axolotl + 4 · model libs: 8 · inference: 6.
Whole populations, 2026-10-17.
Shape and config, the two classes a structural read speaks to,
sit in the model libraries and the serving stacks: where
adapters, quant and configs are edited.
Same five search phrases, four sources:
who reports what
GitHub issues (457)
OOM 67% shape 24%
HF forum (108)
OOM 44% shape 39%
Stack Overflow (117)
OOM 40% shape 57%
PyTorch forum (159)
OOM 27% shape 58%
Bug reporters (GitHub) file memory; question askers (forums,
SO) file shapes. The five phrases themselves push OOM from 19%
to 67% on GitHub, so read these columns against each other,
not against the population study.
Left: the 47-repository population study, 2026-10-17 (regex lower bound = config errors + checkpoint-load shape mismatches, before precision correction). Right: 2026-10-16 replication, Discourse and Stack Exchange APIs; own regexes, nothing hand-audited. Scripts and raw responses kept.
Backup · who makes the change
In ML repositories, agents opened 1.5% of pull requests a year ago. Now 16%.
Agent-signed share of pull requests, 931 ML repositories
Jul
1.5%
Aug
Sep
Oct
Nov
Dec
Jan
Feb
6.8%
Mar
Apr
May
9.2%
Jun
Jul
Aug
Sep
16.2%
Oct
2025 2026 (Oct to the 17th)
What changed
More of the code that asks for GPUs is written, and launched, by an agent. The same repositories went from 5,000 to 29,000 pull requests a month.
What did not change
Each change still needs someone to decide it is safe to run or serve, whoever wrote it. The checks on it still happen after the GPU is taken.
Neurarch study, 2026-10-17: 319,799 pull requests since 2023 in 931 ML repositories, agent signature by co-author or body line (almost all Claude Code). The growth curve is the market's, not ours.
Backup · what leaves the machine
Nothing a competitor could use, and a buyer can run it with the network cut.
Surface Leaves the machine Never leaves
neurarch-trace, Action, bot Layer types, shapes, hyperparameters, a structural fingerprint Weights, source code, data
Opt-in corpus row Type histogram, edge count, rule ids and severities Names, parameter values, the graph
Offline container Nothing. Proven by a test that stubs every socket before import Everything
docs/DATA_POLICY.md is public and changes with the payload. The container's parity with the hosted API is a test over every template.