Shopify CTO Mikhail Parakhin on the AI Phase Transition and SimGym

Summary

• Shopify says internal AI use has effectively reached saturation, with daily active participation approaching 100% after a December 2025 “phase transition.” CLI agents such as Claude Code, Codex, Pi and Shopify’s Rover are outgrowing IDE tools, while employees receive effectively unlimited tokens and are told, “Please don’t use anything less than Opus 4.6.”
• Shopify’s key operating insight is that token consumption is directionally useful but insufficient by itself without high-end critique and PR review. Parakhin calls many noncommunicating parallel agents “almost useless”; he would rather spend slower GPT-5.4 Pro or Gemini Deep Think tokens on verification because models may write code with fewer bugs on average than humans yet produce so much code that aggregate defects rise.
• AI code volume has shifted Shopify’s constraint from generation to integration: PR merges are growing about 30% month over month versus 10% before, while estimated complexity also rises. Longer review is systemically cheaper than failed tests and rollbacks, and Parakhin says human-era Git and CI/CD may need “a different metaphor”—even prompting this lifelong microservices opponent to reconsider independently deployable services.
• Tangle and Tangent turn Shopify’s ML infrastructure into a compounding experimentation network rather than a collection of notebooks. Content hashes prevent duplicated work across teams and carry the same pipeline into production; auto-research then moved search from 800 to 4,200 QPS on the same machines at equal quality, making “the price of electricity” a substitute for hundreds of manual trials.
• SimGym’s defensibility, in the episode’s clearest investor call, comes from decades of merchant and buyer outcomes rather than generic agent prompting. Shopify spent almost a year working toward 0.7 correlation with add-to-cart results, now uses multimodal browser simulations to recommend conversion changes for a single site, and sees users rising daily; the immediate constraint is paying for models and browser farms.
• The larger ambition is to model customers and entire merchants as trajectories, then test when and how interventions change their futures. HSTU-based rollouts can compare coupons, thank-you cards, campaign settings or cashback against no intervention; swyx argues that this temporal, context-rich layer exceeds static A/B summaries, while Parakhin calls it “such an unlock.”
• Shopify is also attacking inference cost and distribution: Liquid models already power a 300M-parameter search system under 30 ms and 7–8B-parameter distilled batch workloads, while UCP opens catalog discovery at runtime. Parakhin says Liquid is taking internal share from Qwen in these niches—not a frontier-model replacement for GPT-5.4—and the catalog plus identity linking exposes “almost a superset” of goods sold worldwide.

Deep dive

1. Shopify crossed an internal AI phase transition in December 2025

• Parakhin’s chart measures “daily active workers,” or employees using at least one AI tool each day, as a percentage of Shopify’s workforce. The aggregate is approaching 100%: “It’s hard to do your job now without interacting deeply with at least one tool.” Incremental model improvements accumulated until a December 2025 “phase transition,” after which adoption and token consumption accelerated sharply.

• The tool mix is changing alongside usage. CLI-oriented systems that do not require employees to inspect code directly—Claude Code, Codex, Pi and Shopify’s internal Rover agent—are growing fastest. GitHub Copilot and Cursor are “not exactly shrinking,” but their IDE-centered workflow is gaining users more slowly.

• Employees can choose their tools, and management focuses on getting the job done rather than enforcing one interface. Shopify effectively funds unlimited tokens while setting a quality floor “from the bottom, not from top”: “Please don’t use anything less than Opus 4.6.” Some employees use GPT-5.4 Extra High, with separate trade-offs around a full one-million-token context window.

2. Token budgets only work when critique scales with generation

• swyx tests Jensen Huang’s rough framing that a $200,000 engineer consuming less than $100,000 of tokens may be underutilizing agents, while noting the obvious “cake seller” incentive and the resemblance to measuring engineers by lines of code. Parakhin nevertheless calls Jensen “directionally correct,” though neither treats a particular token target as established.

• Shopify’s trial-and-error lesson is that raw consumption can hide a damaging anti-pattern: too many parallel agents producing work without communicating. That setup is “almost useless” compared with fewer agents that burn tokens efficiently in a structured loop where one model produces, another—ideally a different model—critiques, and the first revises.

• Those critique loops feel slow because users wait while models take turns, but Parakhin says the resulting code is substantially better. His quality argument is aggregate rather than per-line: a good model now writes code with fewer bugs than the average human, in his view, yet writes so much more that more total defects can still reach production.

• The metric he wants is the ratio between generation spend and expensive review spend. PR review is where Shopify wants GPT-5.4 Pro, Gemini Deep Think and prolonged turn-taking, not a large swarm; Parakhin says standard Codex or Claude Code usage “is not gonna cut it” for stemming production bugs. Finding no satisfactory commercial implementation, Shopify currently uses its own review tooling.

3. Machine-speed coding is breaking human-era CI/CD

• Shopify’s displayed PR merge growth has risen from roughly 10% to 30% month over month, with estimated complexity increasing too. swyx reads that as productivity, but Parakhin emphasizes the operational bill: more code raises the chance that at least one test fails, forcing teams to identify and evict the offending PR, retest without it and restart deployment.

• A review model thinking for an hour can therefore reduce total delivery time. The relevant comparison is not an instant review versus waiting; it is deliberate review versus repeated test failures and rollbacks. swyx summarizes the management lesson: “You don’t look at the individual, look at the aggregate and look at the change in the aggregate system.”

• Shopify uses Graphite and stacked PRs, yet Parakhin calls CI/CD and repository interaction its clearest current bottleneck. Systems designed around human coding speed are “creaking” under agent output and probably need “a different metaphor, or a completely different design.” swyx likens merge conflicts to a global mutex; Parakhin, a “lifelong opponent” of microservices, wonders whether independently shipped services might make a comeback, but stresses, “I don’t know.”

4. Tangle eliminates the digital archaeology between experiments and production

• Tangle targets the familiar lifecycle in which a data scientist cleans missing values, filters bots, mutates scripts, loses a previously working result, then throws a notebook over the production wall. Six months later, paths may have changed or the script may crash, leaving the original author to conduct “digital archaeology” on work they once understood—multiplied across teams and new hires.

• Parakhin distinguishes Tangle from Airflow: Airflow excels when an established job must run repeatedly on a production schedule, while Tangle centers on teams—or agents—developing, sharing and cheaply varying experiments. A user can clone a large pipeline, alter one component, run ten experiments or hyperparameter optimization, and ship the resulting workflow to production “in one click” without porting it elsewhere.

• He calls Tangle a third-generation system incorporating lessons from Ether, Yandex’s Nirvana and Airflow. Components can use any language and reduce underneath to CLI calls plus YAML; users can visually assemble, clone and share them while retaining static versioning and reproducible results.

• Content-addressed caching creates the larger advantage. If a version differs but the output remains identical, nothing reruns; if ten experiments need the same preparation, it executes once without coordination. The bigger saving comes when an unknown team later performs equivalent work: Tangle detects the common output automatically, making the second experiment appear to “jump forward.” That is infrastructure with a cross-team network effect, not merely a faster personal tool.

5. Tangent turns measurable work into an autonomous hill-climb

• Tangent layers an auto-research agent over Tangle. It inspects a pipeline, proposes changes, runs experiments and continues modifying the work against a loss function or other measurable goal. It can rearrange existing components superficially or inspect code and create new CLI-and-YAML components at a deeper level.

• Boris Power reports Shopify search moving from 800 to 4,200 QPS at the same quality, on the same machines, through repeated code optimization. Other loops improved Liquid-theme latency and prompt-compression gisting; one storage investigation also exposed a huge table whose job was effectively hashing one random ID into another random ID.

• The tooling is unexpectedly democratizing. Although ML and AI engineers adopted it first, one of the heaviest users is a PM whose domain and product knowledge can now drive experiments without manually changing code. Parakhin’s shorthand is “Claude Code for your AI development situation”: users specify desired results and iterate without understanding every underlying algorithm.

• Boris Power’s reported limit is equally clear: auto-research handles “kind of obvious things” that humans lack bandwidth to try, not deeply out-of-distribution inventions. A weeks-long hobby run attempted more than 400 experiments and found one improvement. Boris says his own batting average might be higher, but 400 manual trials would take perhaps three years; here, “the price of electricity did it.” He began hoping to prove Karpathy’s optimization example wrong and finished as a supporter.

6. SimGym works because Shopify can calibrate agents against reality

• The objection to simulated customers is the obvious one: agents prompted in a vacuum simply reproduce the prompt. Tobi initially made the same challenge. Parakhin’s answer is Shopify’s decades of evidence linking real storefront changes to sales outcomes—historical information a startup simulator would not possess.

• That evidence is noisy because changes rarely occur as isolated A/B tests; they are often “A/A” comparisons across different periods. Shopify aggregates observations, applies denoising and collaborative-filtering-like methods, and spent almost a year optimizing toward an internal target of 0.7 correlation with outcomes such as add-to-cart events.

• Realistic testing also requires multimodal agents acting through browsers. HTML alone misses visual friction: designers may expect larger product images to improve sales, while Parakhin says sales “usually” tank. SimGym therefore needs many expensive simulations, rendered pages and strong multimodal models. An earlier architecture used GPT-OSS; the current version also runs multimodal models.

• Shopify initially built a system comparing two proposed designs, reflecting the large-company assumption that merchants already had A and B. The team then realized most merchants have only one site and need the alternatives generated for them. SimGym now predicts conversion behavior, suggests modifications and is adding users every day; Boris Power says the immediate problem is lowering model and browser costs through distillation and infrastructure optimization.

7. Counterfactual trajectories extend simulation beyond A/B testing

• Existing merchants improve SimGym because Shopify can recreate the distribution of their actual customers rather than generic personas, dramatically increasing correlation with add-to-cart or conversion behavior. swyx calls that historical feedback loop the shareholder-level moat; Parakhin agrees he would not attempt the same startup without proprietary data because otherwise “whatever you say in prompt” determines the agents’ behavior.

• swyx’s statistical pushback is that an A/B result summarizes endpoints but does not capture context-dependent journeys through time. Parakhin says Shopify already operates an HSTU-based system modeling whole companies and their possible paths. At any point, it can insert a counterfactual intervention—coupon, discount or personal thank-you card—and roll the trajectory forward with and without it.

• The same machinery can optimize both what intervention to make and when to make it. Shopify pals may alert a merchant that Canadian sales appear misconfigured or recommend a campaign with particular parameters; at the buyer level, the platform can model discounts or cashback. Parakhin’s framing: simulating complex humans and companies this way was once dreamed of but “completely impossible.”

• Behavior also varies materially across stores and product categories, requiring aggregation at multiple levels. Farhan Thawar says Shopify recently resurrected the Chinese restaurant process for naturally growing clusters, a method he had not encountered since 2001, to study how buyer modes and sensitivities differ by category.

8. Liquid models are winning Shopify workloads where latency dominates

• Parakhin and the Shopify team frame state-space models as faster, lower-footprint alternatives to transformers with effectively infinite context if their state retains all necessary history, but say they never became expressive enough. Liquid neural networks are “souped-up SSMs”: more expressive and harder to implement, with subquadratic context scaling and particularly strong hybrid forms combining Liquid components with transformers.

• Shopify’s sharpest latency case uses a roughly 300M-parameter Liquid model for query understanding. It expands a typed query into meanings, synonyms and personalized possibilities based on prior searches, end to end in under 30 milliseconds. Achieving that required Liquid AI, CentML and NVIDIA to optimize a stack whose CUDA assumptions were largely designed for much larger models.

• At the opposite extreme, Shopify distills large teachers into roughly 7–8B-parameter Liquid models for high-throughput batch work. Billions of products need taxonomy placement, normalized attributes and identification of equivalent listings across merchants; latency matters less than total throughput and cost. Sidekick Pulse and other long-context workloads use similar task-specific distillation.

• Parakhin is explicit that Liquid is “not a panacea” and will not suddenly compete with GPT-5.4 as deployed today. Yet he calls it the only non-transformer architecture he has found “genuinely competitive” and says hybrid Liquid models may be “the best architecture I’m aware of, period.” His counterfactual is hedged: with Anthropic-, Google- or OpenAI-scale compute, Liquid might compete with or beat the largest models; meanwhile it is taking Shopify workload share from Qwen on merit.

9. Shopify is extending the stack from universal catalog access to personality

• The UCP release discussed at recording adds search across Shopify’s catalog, lookup by specific ID and bulk retrieval, letting product selection be decided at runtime for what to display, sell or check out. Shopify is investing in both personalized and non-personalized search to expose a universe Parakhin describes as “not exactly, but almost a superset” of products sold worldwide.

• Identity linking is intended to reduce friction by connecting Google or other identities to that commerce layer. swyx’s platform conclusion is that no individual merchant could reproduce Shopify’s simulations, inference infrastructure and historical learning alone; Parakhin embraces the framing as “creating network effect.”

• The Sydney retrospective shows that deliberate AI personality predates the current stack. Sydney first shipped in India, initially using a tuned Megatron-Turing model from the Microsoft–NVIDIA collaboration rather than OpenAI, and went largely unnoticed. Its famous edginess was not purely emergent: drawing on Yandex’s Alice, the team intentionally learned that an assistant should be “polite, but…a little bit on edge.” Original Sydney development lead Andrew McNamara now heads Shopify’s Sidekick.

• Shopify’s hiring needs follow the bottlenecks: more ML and data-science specialists to “munge” its expanding data, plus distributed-database engineers. Parakhin says Shopify is working with Yugabyte and exploring whether LLMs can reimagine distributed databases—another part of the broader shift from adding AI features to rebuilding core systems around machine-scale intelligence.