Same Model, Different Shell, 4.5 Point Gap: Model and Shell Are Actually the Same Product
Deep Thought, on AI and Aspirations —— ByteDance Deep Thought Circle
On public evaluation leaderboards for terminal programming models, there’s a counterintuitive ranking: the same Claude Opus 4.6 scores 79.8 points with the ForgeCode shell but only 75.3 with Capy. The weights are identical, not a single byte changed, yet there’s a 4.5 point difference. On leaderboards where every 0.1 point matters, this is nearly a generational gap.
An even more dramatic example comes from Cursor. Their team publicly discussed how the same model, with unchanged weights but a different shell, jumped from outside the top thirty to the top five on the leaderboard.
Engineer Nicolas Bustamante did a thorough breakdown of this phenomenon, dissecting the internal implementations of several mainstream programming agents layer by layer. His conclusion: the model doesn’t exist in isolation—the shell is part of the model. This judgment deserves serious consideration from everyone building AI products.
Models Are Trained for Specific Shells
First, let’s explain why changing shells causes performance drops.
Models aren’t just post-trained against generic APIs—they’re post-trained for specific shells. Every detail in the shell gets baked into training and becomes part of the model’s instincts: what tools are called, what the input schema looks like, how citation tags are formatted, where skill files go, what format the planning protocol uses.
Pull the model out of its familiar shell and performance loss occurs—the kind you can’t recover. This is why every agent that claims to be “model-agnostic” hits the same wall: you can’t just swap models. To swap cleanly, you have to swap the shell too—the tool surface, schema shape, memory format, citation contract, system prompt structure, a whole chain of things must change together. Mismatches won’t throw errors directly; they manifest as quiet degradation: missed tool calls, incorrect reasoning strength, things that should be remembered aren’t.
Tear apart several mainstream shells and you’ll find they’re not three ways to write the same thing—they’re three completely different protocols. One model is trained to “send commits, read events,” another is trained to output tool calls directly in conversation, a third is trained to spawn subprocesses and act as supervisor. Models learn the precise shape of the protocol. Change the protocol and their instincts fail.
A Six-Character Tag That Tells the Whole Story
The most telling example is a small citation tag: oai-mem-citation.
In one shell’s design, every time the model uses a memory, it attaches this XML block at the end of the message, telling the shell “I referenced this memory.” The shell parses it and increments the usage count for that memory. With citations, memories survive; without long-term citations, memories are treated as useless old data and cleared out. Through these six characters, model and shell complete an unspoken collaboration: you tell me which one you used, I reward it.
Switch shells and problems appear immediately. A model with citation habits moves to a shell that doesn’t parse this tag—it spits raw XML directly to the user, and the memory system receives no signal that anything was “used.” Conversely, a model that doesn’t attach tags moves to a shell that depends on them—it’s using memories just fine, but the system thinks these memories have never been used and quietly phases them out within weeks according to decay rules. Same weights, same memories, different shell—one path improves with use, the other slowly rots.
The situation with skill files is similar. On the surface everyone uses SKILL.Md with nearly universal formats. But what a skill file truly depends on are the tools named in its body: a skill that says “Step 1: call TodoWrite” will silently fail in a shell without that tool, or run a crippled version. Between “universal format” and “consistent contract” lies an entire invisible system of tool semantics.
How Honest Products Are Built
Facing this coupling, teams that work honestly share a common approach: stop pretending the shell is neutral, and feed the right dialect to the right model.
One multi-model shell provides a reference implementation: different tool surfaces for different model families—models trained with patch-format file editing only get patch tools, models trained with string replacement only get replacement tools; some models get lazy-loaded tool loops, others get all tools displayed at once; even the review stage uses complementary models to catch each other’s errors. The cost is honest admission: the same product running Claude versus GPT is two different products—there’s no common denominator in between.
The Cursor research team’s recommendation is equally direct: unless there’s a special reason, use only one model per conversation. Mid-conversation switching causes three things simultaneously: conversation history becomes an unfamiliar distribution for the new model, all cached context invalidates, and the tool table switches to a different shape. If you really need another model, the cleanest approach is to dispatch a sub-agent, not switch the main conversation. Sub-agents open fresh context without old conversation bias, standing on their own tool surface from the first turn.
What This Means for AI Product Builders
Translating this into product judgment has three layers.
First, stop using “model-agnostic” as a selling point. Multi-model strategy is a real competitive dimension, but it’s not a free option. Switching between models carries a hidden migration tax: performance loss, tool refactoring, memory loss. “Switching models is just changing an API” only sees the invoice, not the cost.
Second, treat the shell as product strategy, not infrastructure. Every component in the shell essentially encodes an assumption that “the model can’t handle something on its own”: reminding it to verify first, helping it compress context, giving it fixed memory rituals. These assumptions expire as models mature. One vivid observation: scaffolding built for last-generation models becomes dead weight on new models and must be entirely removed. To stay at the capability frontier, when a new model releases, you’ll likely need to delete most of your existing code. This is how this industry works, not an accident.
Of course there are counterarguments. Protocol standardization is advancing, tool calling and skill formats are converging, coupling will decrease long-term; some of the score differences on leaderboards may be benchmark-specific tuning, not entirely real capability gaps. These objections are valid. But the directional truth hasn’t changed: models and the product forms that carry them are growing into parts of each other. Every tool name you define today, every rule you use to tame the model, becomes part of your product’s irreproducibility and the most expensive part to change later. Treat them as strategic design now.
Key points: Same weights with different shells differ by 4.5 points; Cursor jumped from outside top thirty to top five by changing only the shell; models are post-trained for specific shells, mismatches manifest as quiet degradation; memory citation tags determine memory survival, universal format doesn’t equal consistent contract; honest approach feeds the right dialect to the right model; shells encode “model weakness assumptions” that expire, new model releases often require deleting most old code.