Your Model ID Is Not a Lockfile: Silent Updates and the Reproducibility Crisis

You pinned a dated model version. You ran your evals. You shipped. Here’s why the thing answering your API calls next month is still not the thing you tested, and why no provider will sell you the fix at any price.

By Abhimanyu Singhal · July 24, 2026 · 12 min read

Look at a modern model ID. gpt-5-2025-08-07. claude-opus-4-8. There’s a date baked into the string, or a version number, or both.

That string is designed to feel like a lockfile entry. Every other dependency in your stack has one. Docker images pin by digest. Node pins by lockfile. Python pins by requirements hash. Pin the version, freeze the behavior. That’s the contract software runs on.

The model ID looks like it joined that family. It didn’t.

A pinned model ID freezes one layer of a four-layer artifact. The other three layers change without notice, without a version number, and without any field in the API response that would tell you. And the one layer you did pin comes with a contractual expiry date.

This post is about those three unversioned layers, the receipts proving they move, and what to do about it.

One pin, four layers

What you pinned: the weights. The frozen numbers that make the model the model. On providers that honor dated snapshots, these genuinely don’t change after release.

What determines your output: the weights, times three other things.

The serving stack. The routing logic, compilers, numerical precision, and GPU kernels that turn weights into tokens. It changes constantly. It has no version number anywhere.

The hardware and batch dynamics. Which machines ran your request, and whose traffic got batched with yours. This changes per request. It is not versionable even in principle today.

The decoding surface. The sampling knobs and their defaults. These change across model generations, and lately the trend is removal, not refinement.

“It’s the same model” is a claim about layer one. Your output is a product of all four. Everything that follows is evidence for that gap.

Quality changed. The version didn’t.

The best receipt in this space comes from a vendor, on purpose. In September 2025, Anthropic published a postmortem of three overlapping infrastructure bugs that degraded Claude’s output quality across August and early September.

Read the list of causes. A routing error sent short requests to servers configured for 1M-token contexts. A misconfiguration on TPU servers corrupted output tokens, surfacing as stray Thai characters in English answers. A compiler bug made an approximate top-k operation return wrong results, but only at certain batch sizes.

Not one of these touched the weights. Not one changed a model ID. At the peak, 16% of Claude Sonnet 4 requests were degraded. Roughly 30% of Claude Code users got at least one bad response.

Three details matter more than the bugs themselves.

First, users found it before the vendor did. Weeks of “Claude got dumber” reports came in, and were widely dismissed as vibes. The provider’s own monitoring didn’t catch it. If the company running the servers couldn’t see the degradation from inside, your only real instrument is your own evals.

Second, the same model degraded differently depending on where you bought it. First-party API: 16% at peak. The same model on Amazon Bedrock: 0.18%. On Google Vertex: under 0.0004%. Same weights, three different bug exposures, because each channel runs a different serving stack.

Third, Anthropic felt the need to state, in writing: “We never reduce model quality due to demand, time of day, or server load.” I take them at their word. But notice why the sentence had to exist. From the outside, nothing distinguishes a policy from a bug. The customer has no instrument that can tell them apart.

To be clear, publishing that postmortem was an act of unusual transparency, and Anthropic deserves credit for it. That’s exactly why it’s valuable. It is the one detailed public record of something every provider experiences: model quality lives in the serving stack, and the serving stack ships silently, every week, on every provider.

Same weights, same prompt, 80 different answers

Set temperature to zero. Send the exact same request 1,000 times. How many distinct answers should you get?

One, in theory. Thinking Machines ran this experiment in September 2025 against a large open model and got 80 unique completions. The first 102 tokens were identical in all 1,000 runs. Then one token flipped, and everything after it diverged.

The cause is not “GPUs are random.” Individual GPU operations are actually deterministic run to run. The cause is that your request does not run alone. Inference servers batch incoming requests together for efficiency, and the batch size depends on how much traffic happened to arrive at that moment. Different batch sizes make the hardware add up numbers in a different order. Floating-point addition cares about order: (a + b) + c does not exactly equal a + (b + c). The tiny rounding differences occasionally flip a token choice, and one flipped token rewrites the rest of the answer.

Here’s the plain version. Your answer depends on how crowded the kitchen was when your order came in. Same recipe, same chef. Different result at the lunch rush.

This means your output depends on other customers’ traffic. That is layer three: unversioned, unloggable, changing per request.

The kicker: this is a solved problem, and nobody sells the solution. Thinking Machines showed that batch-invariant kernels produce 1,000 identical completions out of 1,000. The cost is roughly 2x on performance. Which is presumably why no commercial API offers it. Check every provider’s pricing page: there is no “reproducible inference” tier. You cannot buy determinism at any price.

The decoding knobs are drifting the same direction. OpenAI’s seed parameter was always documented as best-effort, “mostly deterministic.” Anthropic has gone further: on its newest models, temperature, top_p, and top_k are deprecated outright, and setting them returns an error. The advice is to steer with prompting instead. Whatever the reasons, the direction of travel is clear. As the stack matures, the reproducibility controls are being removed, not strengthened.

Sometimes the ID itself moves

So far the pin held and the layers under it moved. Sometimes the pin doesn’t hold either. There are three distinct behaviors here, and the distinction matters, because they carry different levels of blame.

Drift by design. Google’s model lifecycle documentation is explicit that auto-updated aliases “always point to the latest stable model.” Azure lets you set deployments to auto-upgrade when a new default version ships, and guarantees at least two weeks’ notice before the default changes. If you call an undated alias, you consented to receiving new weights on the provider’s schedule. Most teams made that choice by not noticing they were making it.

The pin that lied. In May 2025, Google silently redirected the gemini-2.5-pro-preview-03-25 endpoint to a newer model. Read that ID again. The date is in the string. Developers discovered it themselves, reported reasoning regressions and requests whose thinking time blew up from 20 seconds to five minutes, and filed a forum post titled “a serious breach of developer trust.” Google formally responded eight days later. It was a preview endpoint, and Google’s policy for stable versions is better. But for that window, every third-party benchmark of that model measured weights that didn’t match the label. Nobody who published those numbers knew.

The named product that swings. The founding study here is Chen, Zaharia, and Zou (2023), who measured GPT-4 in March and again in June of that year. On their code generation test, directly executable output fell from 52% to 10%. The methodology drew fair criticism (formatting changes explain part of it), and those snapshots were labeled, not silent. But the magnitude finding stands: the same product name, one quarter apart, moved more than most model-versus-model eval gaps.

The pattern across all three: the identifier is a marketing surface, not a cryptographic commitment. A docker digest can’t point at different bits. A model ID points wherever the provider’s routing table says it points.

Even the frozen models die

Suppose you do everything right. Dated snapshot, honored by the provider, evals green. Your reproducibility now has a countdown timer attached.

Azure starts an 18-month retirement clock on models at general availability, and the docs state the dates “aren’t extendable.” A retired model returns HTTP 410. Gone. OpenAI’s policy floor is six months’ notice for GA models, and as little as two weeks for previews. In one wave on October 23, 2026, gpt-3.5-turbo, gpt-4, o1, and o3-mini all shut down. Anthropic’s floor is 60 days, and its deprecation table is a steady drumbeat: the flagship of mid-2025 retires in mid-2026. Google blocks access a month before retirement, and some Gemini models retire about 12 months after release.

There’s even a channel wrinkle. Anthropic’s docs annotate some models “retired except Bedrock” or “retired except Google Cloud.” The same model has different death dates depending on where you buy it. Your cloud contract quietly decides how long your pinned model stays callable.

The macro trend is worse. An MIT CSAIL study of 108,514 citations across 62 model variants found the scientific lifespan of models is shrinking 23% per year. Models released before 2020 stayed in active use for four to six years. Post-2022 models peak within one or two. The paper’s phrasing is exact: deprecation “breaks pipelines, severs reproducibility, and forces costly migration to a successor whose outputs may not be comparable.”

Call this what it is. Reproducible until the retirement date is not reproducibility. It’s a lease. And there is no archival tier: you cannot pay OpenAI to keep gpt-4 callable for your audit trail, your regulatory submission, or your published paper. When it’s gone, every result built on it becomes uncheckable. This, incidentally, is the strongest practical argument for open weights, well ahead of the cost argument: a model you can download is reproducible forever.

The regression that wasn’t in your code

Let’s make it concrete. A team ships an agent on a pinned dated snapshot. Their eval suite has 200 tasks. On release day it scores 94%. Here are four ways that number stops being true, with nobody changing anything.

The bug window. A serving-stack incident like Anthropic’s August run degrades some percentage of requests for three weeks. Any eval run inside the window measures the bug, not the model. If you ran your suite once in August 2025 against Sonnet 4, there was a real chance part of your score came from misrouted servers.

The noise floor you never measured. That 94% came from one run. With 200 tasks, the statistical wobble on a 94% score is about 1.7 percentage points either way, before you even add batch nondeterminism. So a drop to 92% on the next run is indistinguishable from noise. To reliably detect a 2-point regression, you’d need roughly ten times the samples: run the suite ten times and compare distributions, not single numbers. Almost nobody does. Which means almost every team’s “the model got worse” claim, and almost every provider’s “nothing changed” reply, are both unfalsifiable on the evidence either side holds.

The consented swap. The team’s Azure deployment was left on auto-upgrade. A new default version ships with two weeks’ notice inside a sprint. The suite now scores 91%. The commit log shows nothing, because nothing in the repo changed.

The forced march. Eighteen months in, the retirement notice lands. The successor model scores 89% on the suite. Now comes the question with no answer: did the successor get worse, or were three generations of prompt tweaks quietly overfit to the dead model? You can’t tell, because the baseline you’d need to test against returns 410.

Your eval number was never a property of the model. It was a property of a tuple: weights, stack, traffic, date. You recorded one element and assumed the other three.

If you sign the checks (business leaders)

Your auditors assume the system of record is stable. Your AI vendor guarantees the opposite, in writing: non-extendable retirement clocks, 60-day floors, 410 Gone. If a vendor sells you an “AI audit trail,” ask one question: what happens to the trail when the underlying model retires?

And budget for the treadmill. Forced migration means re-validating your AI systems every 12 to 18 months, forever. That’s not a one-time integration cost. It’s a subscription you signed without seeing the line item.

If you draw the diagrams (architects)

Version the tuple, not the model string. Log the model ID, the channel, the region, and the API version on every call. Then accept that the stack has no version field you can log, and compensate: capture request and response pairs for a golden set continuously, because stored outputs are the only durable record of what the system used to do.

Treat migration as a standing capability, not a project. The retirement clock makes model migration a recurring operation, like certificate rotation. Architectures that assume the model is permanent are architectures with a scheduled outage in them.

If you write the code (senior engineers)

Stop treating eval runs as point measurements. A single-run score is a sample from a distribution you haven’t characterized. Run your golden set on a schedule, not just at release. Store score distributions, not pass/fail booleans. Alert on distribution shift.

Do the power math once. If your suite is small, compute how large a regression you can actually detect at your sample size, and put that number in the runbook. It will be embarrassingly large. That’s the point.

And re-run everything on every model change, including the ones you didn’t ask for. Your measurements of a model’s usable context, its latency profile, its cost per task: none of those are stable across versions either. A model swap resets every empirical number you’ve collected about the old one.

The honest caveat

Dated snapshots from Anthropic and OpenAI do appear genuinely frozen. The documented silent swaps hit preview endpoints and undated aliases, not honored snapshots. Providers have improved: postmortems get published, notice periods are formalized, Google tightened its stable-version policy after the May 2025 incident. And for a typical product feature, drift within a snapshot is smaller than the variance you introduce every time you edit a prompt.

So the crisis is real but specific. It bites wherever outputs must be comparable across time: eval suites and leaderboards, regulated audit trails, published research, long-lived agents with stored trajectories. If nothing you build needs to agree with its own past, this post is a vaccination, not a diagnosis. If any of it does, the infrastructure is working against you, by design.

The number nobody prints

Every layer of your infrastructure carries a version number except the one that generates the output. There is no field in any API response that changes when the provider swaps a compiler, a kernel, a precision setting, or a routing table. OpenAI’s system_fingerprint was the one attempt at such a field, and its own documentation conceded it changes when the backend changes, which is another way of saying the backend changes.

That missing version number is the whole story. The industry adopted the costume of versioned software: the dates in the IDs, the deprecation tables, the changelogs. It skipped the substance, which is that a version pin is supposed to be a promise about behavior, all of it.

You can pin the weights. Nobody will sell you a pin for the machine they run on.

References & further reading

The serving stack moves under a fixed model ID

Nondeterminism at temperature zero

Behavior change under a stable name

Versioning, retirement & lifecycle policy

The reproducibility cost

← All posts