N
Neurarch

The dependency scanner for
open-weight models.

Finds every model your code and deploy configs pull, shows what changed upstream since the commit you approved, flags what is a security risk, and blocks the upgrade in CI.

~/your-repo
$ npx neurarch scan .
1 file read, 1 model reference, 1 high, 3 medium
nomic-ai/nomic-embed-text-v1.5 @ b0753ae76394 embed.py:3
high security auto_map runs code from nomic-ai/nomic-bert-2048; revision= does
not pin it, so that repository's main is what loads
fix: code_revision=<commit> (scan --fix writes it)
medium behavior context window 8192 to 2048 tokens (2026-04-07)
medium behavior declared dtype float32 to bfloat16 (2024-08-26)

Real output of neurarch 0.4.0 on a one-file repository, 2026-11-18; long lines shortened. Each finding links the Hub commit it came from.

Xin Gao · Founder · www.neurarch.com · November 2026

The problem · one model, read from the Hub

You pinned the model. The code it runs is not pinned.

Your code SentenceTransformer( "nomic-ai/nomic-embed-text-v1.5", revision="b0753ae", trust_remote_code=True) Model repository nomic-embed-text-v1.5 @ b0753ae · 2024-06-05 weights, config, tokenizer: pinned auto_map Code repository nomic-bert-2048 @ main · 2025-04-29 revision= does not reach here nomic-bert-2048: every commit that changed its Python 24 before your pin23 after it: the code that runs today Jan 2024 Jan 2025 your pin 2024-06-05 Jun 2024 main today 2024-01-04: 5 commits changing Python2024-01-31: 3 commits changing Python2024-02-23: 3 commits changing Python2024-03-19: 3 commits changing Python2024-03-29: 1 commit changing Python2024-05-02: 6, 2024-05-03: 1 commits changing Python2024-06-05: 2 commits changing Python2024-08-26: 3 commits changing Python (after the pin)2024-10-01: 3 commits changing Python (after the pin)2024-11-18: 5, 2024-11-19: 1 commits changing Python (after the pin)2024-12-18: 2 commits changing Python (after the pin)2025-02-07: 2, 2025-02-11: 5 commits changing Python (after the pin)2025-02-24: 1 commit changing Python (after the pin)2025-04-29: 1 commit changing Python (after the pin)
17 of 150
popular models that run repository code load it from another repository table ↗
19.3M
downloads of those 17 in the last 30 days
Hub API, 2026-11-17
11 of 17
had that code change since they started pointing at it (18.3M downloads) commits ↗

The problem, in public code · 353 files read by hand

The teams that pin are the ones it catches.

353 public Python files load one of the 17 one square per file; GitHub code search, every file read 17 pin a commit with revision= 336 do not pin at all 167 load through sentence-transformers · 5 mention code_revision A maintainer in 2024 (#2613 ↗), on fixing it: "very few models use third party repositories" On popular models: 17 of 150, 19.3M downloads a month.
MTEB, the embedding leaderboard Pins nomic-embed-text-v1.5 at b0753ae. The code that runs is nomic-bert-2048 at its 2025-04-29 head: 23 Python changes later, including "fix: dtype" and "fix: mask". mteb/.../nomic_models.py ↗ The same model, pinned at its first commit e178409 (2024-02-10) points at a code repo whose main deleted the file. Loading it today: OSError: ... does not appear to have a file named configuration_ hf_nomic_bert.py

The problem, counted · 80 public incidents and the change tracker

Publishers change model files often, and the config shows the cause in 26 of 30 incidents.

Where the change came from, and whether the config shows the cause The model's publisher changed a file (30) template, EOS, tokenizer, config, a new revision 26 readable from the config 4 partly, 0 not An engine, framework or provider changed (50) version upgrades, kernels, quantization loaders, flags 13 19 partly 18 not: kernel and implementation bugs How often the publisher changes it, after release 114 of 435 models with a chat template changed it later, and the rendered prompt changed (26.2%, 2026-11-17); 95 within 30 days. 84 of 672 changed the EOS value (12.5%). First public report: median 4 days after the change; the tail runs 76 to 232 days.
Three that shipped without an error gpt-oss-120b · one line, 76 days <|call|> added to the stop tokens. Parallel tool calls collapsed to one; first reported 76 days later. HF openai/gpt-oss-120b discussion #151 ↗ DeepSeek-R1 · 9 bytes, weights unchanged The template began prefilling <think>. Day 4: reasoning output went null in vLLM. HF deepseek-ai/DeepSeek-R1 discussions #144, #181 ↗ Qwen2.5-7B-Instruct-GGUF · still stale The official GGUF ships the tool template the base fixed on 2024-09-19; 181,341 downloads in 30 days. Read from the GGUF header, 2026-10-19

The problem, security side · public advisories, dated

Model files are an attack surface: 9 public advisories and reports since 2024, 6 of them in 2026.

2024 2025 2026 May 2024 · criticalllama-cpp-python: a GGUFchat template runs codeCVE-2024-34359 ↗ Apr 2025 · criticalPyTorch: weights_only=Truestill runs a crafted fileCVE-2025-32434 ↗ Jan 2026 · highvLLM ran auto_map code withoutchecking trust_remote_codeCVE-2026-22807 ↗ Apr 2026 · critical · CVE-2026-5760 ↗SGLang: a model's template runs code Jun 2026 · mediumvLLM: pinned deploymentsload unpinned codeCVE-2026-47155 ↗ Sep 2025 · Unit 42 ↗deleted author re-registered,model replaced under thesame id; fix: pin a commit Feb 2026 · ICLRtemplate-only backdoor:18 models, 4 enginesarXiv 2602.04653 ↗ May 2026 · high · CVE-2026-4372 ↗transformers: one config fieldloads another repo's code Oct 2026 ↗vLLM: a request'scode_revision runscode

What scan reads in our 608 tracked models: 150 run repository code (17 from another repo) · 166 template changes altered the prompt, each rendered at both commits · 15 ship pickle weights only.

Sourceseach date links its advisory·GitHub Advisory Database·change record

Why now

Open-weight models change more often, and more of each change lands before the tools support it.

New Hugging Face uploads whose architecture transformers, vLLM or llama.cpp did not support that day 8.1% 2024 Q2 16.0% 2026 Q3 HF Hub, Neurarch analysis 2026-10-18; ~35,000 uploads a quarter; class-name match "Wrong output" issues naming a model, 9 inference and tooling repositories, per month ~12 2024 Q1 ~75 2026 Q3 GitHub issues, Neurarch analysis 2026-10-18; title keywords; 60% to 72% true on a sample of 40 Pull requests signed by a coding agent, 931 PyTorch- ecosystem ML repositories 1.5% Aug 2025 16.0% Sep 2026 GitHub, Neurarch analysis 2026-10-17; 319,799 pull requests since 2023 Open-weight models' share of tokens on OpenRouter, one week 72% 100% week to 2026-09-22 OpenRouter, "The token state of play"; closed models still take 73% of spend
SourcesOpenRouter token share·HF Hub and GitHub counts: Neurarch analysis, method in notes

The product · one command, in CI

Every model your code pulls, what changed since your pin, and the fix as a diff.

1 · Find every reference Python calls, and ids heldin variables registry records (MTEB) k8s, compose, Dockerfile env vllm / sglang launch flags MTEB repo: 2 found beforerecords were read, 923 after 2 · Your pin vs today config, tokenizer, template auto_map and the reposit pulls code from weight format the daily change record no weights downloaded,no GPU, no model run 3 · Three lanes, never mixed securitycode on load, pickle, CVE fields behaviortemplate, tokenizer, defaults hygieneunpinned, unreadable, renamed 4 · Out, where it is read exit code that blocks the PR --fix: pins written into code --cyclonedx: ML-BOM, mappedto NIST SP 800-218A OSV advisory feed Renovate PR when a pin moves

before embed.py

model = SentenceTransformer(
"nomic-ai/nomic-embed-text-v1.5",
revision="b0753ae76394dd36bcfb...",
trust_remote_code=True,
)

after neurarch scan --fix

trust_remote_code=True,
+ model_kwargs={"code_revision": "7710840340a0..."},
+ config_kwargs={"code_revision": "7710840340a0..."},
)
re-scan: 0 high. Nothing you run changes today;
it just stops changing underneath you.
Sourcesneurarch on npm·the nomic case·output captured 2026-11-18

Why believe it · what changed, recorded and published

754 recorded changes to 672 open-weight models, in formats other people's automation can read.

754 changes, by the year of the commit 2019: 4 recorded changes420192020: 39 recorded changes3920202021: 10 recorded changes1020212022: 19 recorded changes1920222023: 44 recorded changes4420232024: 261 recorded changes26120242025: 214 recorded changes21420252026: 163 recorded changes1632026 Only models tracked today, so early years are undercounted; 2026 runs to Nov 17. 311 of 672 models changed a tracked file after upload. By what changed tokenizer244chat template220structural config213defaults, metadata, format77
Published asWhat it does
OSV advisoriesAll 754 validate against the official schema; CC BY 4.0
Renovate datasourceOpens a pull request when a pinned model moves
pin-checkUnpinned downloads. Held out, 18 files: 0 missed
scan + ML-BOMThree lanes per model; CycloneDX 1.6, strict-valid, mapped to NIST SP 800-218A

Feeds live at neurarch.com/changes-feeds/; tracker figures as of 2026-11-17. pin-check and scan are on npm, neurarch 0.4.0 (2026-11-18).

Competition · what each one reads, from its own documentation

Scanners judge a file at one moment. None we checked compares your pin with today.

CVE invllm, torch maliciouspayload in a file finds model idsin code + deploy diff sinceyour pin code fromanother repo blocks in CI,writes the fix Package scannersSnyk, Socket, Dependabot Artifact proxiesJFrog, Sonatype (proxy the Hub) Model file scannersProtect AI (Palo Alto), HiddenLayer, HF Lint ruleBandit B615 (unpinned download) Neurarch scanreads all three lanes does it in part not in its docs Neurarch's halves: engine-version CVEs only (vLLM, transformers); pickle format, not payload content

The category already has buyers and exits: Palo Alto Networks bought Protect AI (SecurityWeek); JFrog scans Hugging Face models with Hugging Face (JFrog). We sit next to them: they say a file is malicious, we say it changed under your pin.

Who buys it · and the evidence they already want it

AppSec pays, the ML platform team runs it, and NIST already asks for exactly this check.

ML engineeradds or bumps a model in a pull request neurarch scan in CIblocks on security, comments on behavior, writes the BOM ML platform teamuses it: owns the modellist and the pins AppSec / ProdSecpays: owns the scannerbudget and the policy What an auditor asks for, by name SP 800-218A PW.4.4 verify acquired AI models, including configuration, before use (High priority) PS.3.1, AI 600-1 GV-1.6-003 track model versions each scan finding carries the item it maps to
Teams already build pieces of it by hand NVIDIA Dynamo #11369 ↗pins Hub resolution to the worker's commit exo #2085 ↗upstream dropped a tokenizer file; model broke MTEB ↗pins a commit on 916 registry entries vLLM, 2 security advisories ↗--revision did not reach every file it loads Unit 42 (Palo Alto Networks) ↗its advice for namespace reuse: pin a commit
SourcesNIST SP 800-218A·Unit 42·none of these teams is a customer

What we do not claim

Most changes are fixes, not attacks. The scan says which lane a change is in and never inflates it.

222 template edits, each rendered at both commits 166 changed the prompt 56 cosmetic 0 trigger-shaped (security) We once counted 9; all were Qwen2.5-VL testing a content field. Corrected in public (#528). 18 more edits could not be rendered and are labelled so. 17 code repositories that revision= does not pin 0 with malicious code found. The risk is that the pin does not decide what runs, not that anyone has abused it yet. A first read is a baseline, never an alert.

It never says

  • this model is malicious
  • this template edit is a vulnerability
  • this output is correct
  • we save you X

It only says

  • which models you pull, and where
  • what changed since your pin, with the commit
  • which of those runs code or can execute
  • the one-line fix

Where it is still weak

Three weaknesses, each with a number.

0
outside teams running scan (on npm since 2026-11-18)
outside users: 08 US teams to call
Next: 8 discovery calls; MTEB notified first.
1 hub
Hugging Face only; other registries not seen
Hugging Faceothers: not yet608 models tracked daily
Next: OSV ecosystem proposal for Hub models, drafted.
$0
revenue. No paying customer yet
paid: 0first price: the $3,000 report
Next: the report runs the scan on their repos.
SourcesStatus as of 2026-11-18

Traction

Small, and here is exactly how small. No revenue, and no call yet from a key that is not ours.

17
outside signups, of 24 accounts
no revenue · Supabase, 2026-10-18
1,496
npm downloads of the MCP server, last 30 days
npm ↗ half on 3 release days
291
npm downloads of the neurarch CLI, last 30 days
npm ↗ 10-19 to 11-17
298
machine calls to the check; 0 from a key that is not ours
295 unattributable · 2026-10-18
2%
14-day retention of the free visitor tool
12 of 679 matured first visits
neurarch-mcp · downloads per day, 10-19 to 11-17 neurarch CLI · downloads per day, same 30 days 10-21: 79 downloads10-22: 14 downloads10-23: 355 downloads (release day)35510-24: 30 downloads10-25: 34 downloads10-26: 68 downloads10-27: 24 downloads10-28: 63 downloads10-29: 40 downloads10-30: 23 downloads10-31: 17 downloads11-01: 26 downloads11-02: 32 downloads11-03: 12 downloads11-04: 162 downloads (release day)11-05: 232 downloads (release day)23211-06: 23 downloads11-07: 40 downloads11-08: 25 downloads11-09: 27 downloads11-10: 21 downloads11-11: 21 downloads11-12: 43 downloads11-13: 38 downloads11-14: 12 downloads11-15: 12 downloads11-16: 11 downloads11-17: 12 downloads 10-21: 32 downloads10-22: 12 downloads10-24: 6 downloads10-25: 24 downloads10-26: 4 downloads10-27: 39 downloads3910-28: 8 downloads10-29: 9 downloads10-30: 10 downloads10-31: 7 downloads11-01: 7 downloads11-02: 15 downloads11-03: 7 downloads11-04: 2 downloads11-05: 5 downloads11-06: 10 downloads11-07: 3 downloads11-08: 15 downloads11-09: 3 downloads11-10: 6 downloads11-11: 8 downloads11-12: 3 downloads11-13: 5 downloads11-14: 8 downloads11-15: 14 downloads11-16: 8 downloads11-17: 21 downloads release day: mirrors fetch every new version (749 of the 1,496) the launch burst, 899 on 10-18, is outside this window on purpose A tool a person must reopen 2% came back. The rows it produced are unattributable. We stopped building for this user in September. A scan that runs in CI on every pull request Nobody has to come back: it runs where the model is added. 30-day test, decided by a payment on 2026-11-16.
Sourcesneurarch on npm·neurarch-mcp on npm·accounts and calls: our database, dated on each tile

Who pays, and how

Entry is a $3,000 report. The subscription is per managed GPU, free under 8.

First buyer

The platform or inference lead who self-hosts open weights on 64 or more GPUs, and both fine-tunes and serves. Every base-model swap, recipe, split and serving config is theirs to approve.

Second segment, to validate

Chip-vendor model bring-up, inference-provider onboarding, and model labs before release. They already pay for day-0 support (evidence in notes).

Free under 8 GPUs; the CLI and the MCP server are free $3,000 report, 30 days, flat requested vs actual over their jobs, plus a pre-check of every change they made; credited to a subscription $15 per managed GPU a month billed annually ($180 a GPU-year), no minimum; Enterprise $25 adds SSO and supported self-host Comparable: Run:ai lists $2,600 per GPU a year (AWS Marketplace). It schedules the GPUs; we decide before a change takes one.
SourcesPrices decided 2026-10-18; none paid yet; detail in notes

Market, bottom up · at our list price per GPU

At $180 per GPU a year: $0.8B this year, doubling by 2030 with the fleets.

Fleet size Organizations GPU spend / yr GPUs x $180 8 to 63 GPUs ~14,000 $0.36M $5.8K 64 to 499 ~4,700 $2.7M $40.5K 500 to 4,999 ~1,100 $22M $281K 5,000+ ~140 $300M+ $2.25M Violet edge: where we sell first, the wedge (64+ GPUs); without the 5,000+ that build their own, $0.50B. $0.81B 2026 $1.6B 2030 Wedge 64+ GPU fleets $1.1B 2026 $2.1B 2030 Serviceable + small teams,inference $70Mto $350M 2026 $135Mto $675M 2030 GPU-cloud channel $0.01 to $0.05per GPU-hour 20262030faded = high end of a range
SourcesModelled, not a census; method in notes

What thickens with time · measured, not asserted

Anyone can rebuild the change list from git. Not what we watched happen.

Rebuildable from git history today 749 of 754 change rows (99%): not a moat, and we say so Only exists because we were watching 576hashed weight snapshots; 7 weight changes caught 9.9 hmedian from commit to our record 8,980model-days of download counts, 15 days so far 2commits later erased from main; git no longer has them measure:reconstructible, 2026-11-17; both erased heads in one GGUF repo.
And three that need someone else Read by other people's tools OSV feed, Renovate, ML-BOM, badges today: 0 outside readers The customer's own record every pin, finding and waiver, per repo today: none; starts with the first customer The position in CI a required check; removing it is a policy change today: none; starts with the first install

The risk, said first: Hugging Face shows code-repo commits on the model page, or a package scanner adds model ids. The history before that day stays ours; the race is to be the check teams already run.

SourcesChange record·feeds others can read·counts from the tracker, 2026-11-17

Team · the next 30 days · the ask

One person built all of it. Next: a payment, an 8-GPU measurement, two buyer replies.

Xin Gao, founder

Eight years in industry: machine learning engineer, then lead applied scientist and AI research scientist. PhD. Solo founder; built every box on the product slide.

The ask

A pre-seed round is open; terms in the conversation. Use of funds, each tied to a number on these slides: the first 3 paying organisations, layout and memory calibration on 8 GPUs and more, and a bring-up triage pilot.

1 · Payone paid $3,000 change pre-check report,from 20 self-hosting teams 2 · Layouton 8 x H100 the recommended split is faster,by at least half the prediction 3 · Bring-upa chip vendor or inference provider asks fora second triage report 4 · Config watch3 teams say an alert would change a decision;1 connects its model list 5 · Outside calls5 keys that are not ours call the check (today: 0) The 2026-11-16 decision, written down now 1 closesraise on the changepre-check; 2 to 4are the evidence only 3 signalsfirst buyer becomeschip vendors andproviders; 30 moredays 1, 3 and 4 failwe say so, andchange what we sell
Sourceswww.neurarch.com·the bot, the MCP server, the CLI and every dataset are public

Backup

Backup: the engine under the scan.

These slides are the earlier pre-check story. Every number in them still stands; they are here as proof of the depth behind the scan, not as the pitch.

The problem

A change breaks in places that raise no error. Most of it is readable from the config before launch.

of LLM training jobs: the last pipeline stage (LM head and loss) caused over half of the slowdown ByteDance, OSDI 2025 (arXiv 2505.05713) 39.3% of jobs slowed by uneven sequence lengths, 1.34x on average ByteDance, OSDI 2025; slowed = fwd/bwd time correlation ≥ 0.9 21.4% of 929 inference-engine bugs had a configuration root cause Liu et al., arXiv 2506.09713 (TOSEM) 23% of the 406 most-downloaded HF models with a config changed a structural key after release (vocab, max positions, hidden size, rope) HF Hub config history, Neurarch analysis 2026-10-17 23% Worker problems, hardware or software, caused over half the slowdown in only 1.7% of straggling jobs (ByteDance, OSDI 2025). The rest was the job's own shape.
Same weights, different config gpt-oss-120b, 20 agent tasks, same requests Ollama 0.650 vllm-mlx 0.400 cause: the chat template drifts from the training format in 3 places. vllm-mlx #568 Kimi K2 tool-call schema accuracy official 100% vLLM 76.00% Together 71.96% Moonshot K2 vendor verifier (K2-0905) Nothing crashed. A standard eval does not see it until someone runs the right one.

When it is found today

Today a change is found out after it holds a GPU or serves traffic. The answers exist at submit.

The changeWhat goes wrongFound todayThe pre-check answers at submit
Swap a model or checkpointA structural key changed after release: vocab, max positions, rope, hidden sizeat load, or in evalWhat changed: structure and config diff, before anything downloads weights
New fine-tune or RL recipePeak memory over the card; data longer than the config acceptsfirst batch, OOMWhat it needs: training peak per card, smallest card, cost, data fit
New TP / PP / DP layoutAn illegal split; the last stage carries the head and loss; uneven sequence lengthsin a trace, after the runIs the layout sound: legality, per-stage work and memory, bubble, sequence imbalance
New serving configContext past what rope was trained for; TP against KV heads; template or sampling driftin user reportsDoes it match the reference: the configuration, not the outputs
Sourcesmethod and sources in speaker notes (S)

Demand, in public · GitHub threads, November 2025 to October 2026

Teams write parts of this check by hand, one project at a time.

AnswerThread, openedWhat they built or asked for
What it needsvLLM #29325, 2025-11-24A TP and max-length planner without installing vLLM; closed, not planned
What it needsUnsloth #5225, 2026-04-30Patches its own KV estimate for a new attention pattern
What it needsOllama #17099, 2026-07-09A new release inflated the estimate; the model no longer fit
What it needsFastVideo #1978, 2026-10-18Adds a VRAM calculator to its deployment cookbook
LayoutvLLM #50576, 2026-07-31Last pipeline rank carries the LM head; layers re-split by hand
LayoutvLLM #52435, 2026-08-15Experts not divisible by the EP size: every worker crashes
What changedexo #2085, 2026-05-11Upstream repo dropped a tokenizer file; the model stopped loading
What changedDynamo #11369, 2026-07-07Pins Hugging Face resolution to the worker's commit SHA
ReferenceLM Studio #2012, 2026-06-05Chat template calls an undefined macro; tool calls crash
Sourcesmethod and sources in speaker notes (S)

The product

One typed graph of the change. Four answers, enforced where the job is admitted.

Typed structural graph from config.json, or a trace 195 layer types · 50 structural checks no weights · no GPU · deterministic the same answer to a person, an agent, CI What changedConfig, tokenizer, template, licence: per revision, 472 models.live What it needsTraining and serving peak per card, smallest card, cost, data fit.live Is the layout soundTP/PP/DP/CP/EP legality, per-stage work and memory, bubble.live Does it match the referenceA serving launch: context, TP, KV memory, template, sampling.live Where it is enforced SLURM cli_filter · built SkyPilot policy · built Claude Code hook · live MCP · live API · live PR bot · live offline · built
Sourcesmethod and sources in speaker notes (S)

What a caller gets back · real output, two commands

A last stage 24% slower than the mean, and a context change that no longer fits.

what it needs seq_len: 1024 → 2048

DocLM ⛔ no longer fits A10G 24 GB
Peak training memory 38.4 GB at batch 32 x 2,048
needs 44.2 GB with the 1.15x headroom
vs base +17.8 GB (20.6 GB at 1,024)
largest batch that fits: 16, 2 accumulation steps
smallest card that holds it: L40S 48 GB
commit 2, batch 16 x 2: fits, 20.6 GB

is the layout sound TP 4 x PP 4 x DP 8

Qwen/Qwen3-8B · 128 x H100 80 GB
stage extra vs mean peak/GPU
0 embedding 0.92x 16.8 GB
1 - 0.92x 12.4 GB
2 - 0.92x 11.1 GB
3 head+loss 1.24x 14.8 GB
split 10,10,10,6 brings the slowest stage to 1.02x
derived from the config, not timed on a GPU

Neither change raised an error in review. Both answers existed before the job was submitted, and both are kept with the change.

Sourcesmethod and sources in speaker notes (S)

Why believe it · the layout check and the config read, measured

The last stage was heavier on every GPU run; 68 of 81 parameter counts land within 2%.

LM head + loss against one block: measured / predicted within 30% 1.4 1.2 1.0 0.8 0.6 1.23 1.11 1.28 0.72 0.99 L4, 3 runs H100, 2 compute-bound runs The heavier side was right on all 5. Per-unit times summed to within 1.4% to 2.5% of the whole step. The size varies by card.
Parameters from config.json, no weights read, against 81 comparable Hugging Face checkpoints 68 within 2% 13 not 0.3% median error, published at neurarch.com/m/ Every miss is printed on the same page as the hits. Split A/B on 2 x H100, PP 2 Recommended split beat the even one on every repeat; predicted gain 1.5 points off measured. Not reliable yet Absolute step time, about 24% high on H100. We quote gains, never a predicted step time.

Why believe it · memory: 203 measured runs

Held out, it flagged 113 of the 131 times a run did not fit a card; accelerate, at most 8.

97 runs of 1 GB or more, one dot per run per method 1 4 16 64 ≤0.1 1 4 16 64 measured peak on the GPU, GB (log) predicted, GB (log) Neurarch, held out by architecture accelerate estimate-memory, float32 predicted = measured
Of 131 times a run did not fit a card, said so Neurarch, held out 113 accelerate, float32 8 accelerate, bf16 0 Median error, runs of 1 GB or more 7% held out by architecture 14% on 25 runs measured after the fit The out-of-sample number is the one to hold us to.
SourcesAll 203 runs, predicted vs measured·method in notes

Why believe it · the serving check, against what vLLM itself printed

Out of sample: VL KV memory within a median 5.96% of vLLM's log, CUDA graphs within 0.38 GiB.

Vision-language launches, KV memory: median error 10 public vLLM logs, none used to fit before 9.35% now 5.96% the vision tower's profiling pass is now counted 0.38 GiB CUDA graph memory, mean error against vLLM's printed estimate, 10 logs held out was 0.63 GiB on the same 10 logs 1.07% KV token count error, out of sample End to end on public logs, KV memory: median 7.6% off. The mean is pulled up by a few outliers, so we do not quote it. Nothing here ran on a GPU: every prediction is from config files.

What it reads now

Multimodal configs as the language model. TP, KV heads and length come from the text config, not the vision tower listed first.

Weights against the checkpoint. The predicted size is checked against the published one; a mismatch is a warning, and memory is sized on the published bytes.

New cards. A40, GH200 96 GB and 144 GB, RTX PRO 6000, and Ascend 910B for memory only.

A receipt. After launch, what vLLM measured, printed next to what was predicted.

On npm since neurarch 0.4.0 (2026-11-18).

Sourcesmethod and sources in speaker notes (S)

What we do not claim

A legal design is not a good design, and we publish the number that says so.

0.17
Spearman correlation between our static score and the trained outcome, across 24 GPU-trained designs. Essentially uninformative.

Measured, and published when wrong

An agent edited Karpathy's GPT for 3 hours on an H100 with the check in shadow: 30 changes, 0 refused, 0 crashed, parameter count within 0.25% on every row. The GPU-fit line was wrong on 27 of 28 rows; that failure started the memory calibration two slides back.

So the product never says

"This design will train better." · "We save you X% of GPU spend." · "The model got it wrong." · "This deployment produces correct outputs."

It only says

What changed. How much memory at your batch, whether it fits your card, and what one run costs, with the assumptions. Whether a split is legal and which stage is heaviest. Whether the serving configuration matches the reference: the configuration, never the outputs. And it keeps that answer.

Sourcesmethod and sources in speaker notes (S)

Backup · what breaks in training repositories, counted

100,909 issues from 47 training repos: one in six is a failure, half of those at startup.

Share of 15,486 failure reports: whole populations of 43 repos, keyword samples of 4 giants Other RuntimeError24.5% Out of memory18.2% CUDA driver / kernel13.5% Shape / size mismatch12.5% NCCL / distributed hang9.6% Config error (heads, rope)8.4% dtype / device6.8% NaN / inf6.5% Share a pre-launch structural read could catch 5.5% after hand-checking up to 36% if every memory and shape report is counted
Of 2,112 reports that say when the run died (regex, unaudited) 53% at startup Startup, first step: 53% Early, under 15 min: 18% Hours in: 17% rest: mid-run or vague Inference and diffusion apps die at startup 70% of the time. Cheap in GPU time. Expensive in iterations: each one is a queue slot, a restart, and an agent turn spent reading a stack trace.
SourcesDatasets·method in notes

Backup · why we do not pitch GPU savings

In well-run labs, the failures a static check catches are the fast ones.

Failed GPU jobs, Acme cluster (Shanghai AI Lab, NSDI'24), 152,838 jobs Share of failed jobs that die within 10 minutes 96.4% Share of failed GPU-time those fast failures consume 5.3% Share of failed GPU-time attributed to infrastructure (NVLink, ECC, network), not the model 82%
Sourcesmethod and sources in speaker notes (S)

Backup · the same failure classes, by source and by repository type

On whole issue populations, the failure tail moves by repository type.

GitHub issues by repository type, share of failure issues core fworchestr.fine-tunemodel libsinference CUDA OOM 7.6 9.2 11.3 6.2 12.4 shape 11.5 8.6 13.0 21.1 6.9 NCCL 12.8 21.6 17.2 3.8 7.1 config 7.6 12.2 10.7 8.4 11.3 lower bound 15.114.914.113.111.9 core fw: pytorch, Lightning, torchtitan, fairseq · orchestration: accelerate, DeepSpeed, Megatron, ColossalAI + 3 · fine-tune: peft, trl, unsloth, axolotl + 4 · model libs: 8 · inference: 6. Whole populations, 2026-10-17. Shape and config, the two classes a structural read speaks to, sit in the model libraries and the serving stacks: where adapters, quant and configs are edited.
Same five search phrases, four sources: who reports what GitHub issues (457) OOM 67%shape 24% HF forum (108) OOM 44%shape 39% Stack Overflow (117) OOM 40%shape 57% PyTorch forum (159) OOM 27%shape 58% Bug reporters (GitHub) file memory; question askers (forums, SO) file shapes. The five phrases themselves push OOM from 19% to 67% on GitHub, so read these columns against each other, not against the population study.
Sourcesmethod and sources in speaker notes (S)

Backup · who makes the change

In ML repositories, agents opened 1.5% of pull requests a year ago. Now 16%.

Agent-signed share of pull requests, 931 ML repositories Jul 1.5% Aug Sep Oct Nov Dec Jan Feb 6.8% Mar Apr May 9.2% Jun Jul Aug Sep 16.2% Oct 20252026 (Oct to the 17th)

What changed

More of the code that asks for GPUs is written, and launched, by an agent. The same repositories went from 5,000 to 29,000 pull requests a month.

What did not change

Each change still needs someone to decide it is safe to run or serve, whoever wrote it. The checks on it still happen after the GPU is taken.

Sourcesmethod and sources in speaker notes (S)

Backup · what leaves the machine

Nothing a competitor could use, and a buyer can run it with the network cut.

SurfaceLeaves the machineNever leaves
neurarch-trace, Action, botLayer types, shapes, hyperparameters, a structural fingerprintWeights, source code, data
Opt-in corpus rowType histogram, edge count, rule ids and severitiesNames, parameter values, the graph
Offline containerNothing. Proven by a test that stubs every socket before importEverything
Sourcesmethod and sources in speaker notes (S)