LangChain's Chase: vertical agents win, routing lives in the harness

Summary

• Harrison Chase’s clearest call: routing for vertical agents belongs in the harness, not the gateway, and routing is not a commodity. The harness knows state — prompt cache, prompt strategies, task context — and is his preference for domain-specific routing. Gateways have a portability case and may fit general coding or chat. He estimates most OpenRouter usage comes from being a gateway, not a router; smart routing happens less than people think, and a frontier model that gets better at everything can erode routing’s value.
• If agent value is derived through verification, domain experts win — and Chase is “massively bullish on vertical agents.” Whoever can write verifiers and build evals captures value; practically, founders should start at the context/harness layer and only post-train models once they have users. On Harvey and Ramp “building their own models”: much of it is post-training open-weight models, sometimes Chinese models, which he defends as valuable — “if you’re training a model from scratch, you’re probably at NeoLab.”
• Memory is the biggest unsolved layer in agents, and LangChain has the scar tissue to prove it. The hard part isn’t the database (some companies store memory in “a single file”) but distilling observations into context — a meta-reflection task LLMs still struggle with. LangChain launched and killed a commercial memory product because it “tried to be too general”; memory must be domain-specific, yet he expects “all products in a few years to have some aspect of memory.”
• LangChain spends roughly 30% of engineering salary on coding tools — and spend has gone down while token usage rose, via visibility, budgets, and owning the harness. One misconfigured plugin capturing the DOM every agent turn burned $40K in a week. Harry Stebbings frames this number as the fundamental determinant of Anthropic’s $2T value; Chase agrees it underpins it, but says software engineering is the only domain agents have really affected so far and expects agent impact in other domains to grow substantially.
• His biggest change of mind: “the market for inference is massive” — passing on building a gateway a year and a half ago was “100% the wrong decision.” He invokes Eric Vishria’s broader observation that “everything just works” and the market is big, and recalls Thomas Tunks’s point about inference demand. Chase says Fireworks, AWS, and Crusoe can all work because demand is so large. Enterprise fear of open-source Chinese models is “largely unfounded,” and he expects comparable U.S. open-source models within 12 months.
• The agent-era internet could be agent-to-agent: providers expose their own vertical agents, and horizontal agents interact with them. Chase’s proposed responses to infrastructure strain and lost ad/upsell opportunities are that vertical agents can interact with underlying systems more efficiently, and ads or recommendations could appear in the agent’s natural-language response.
• On venture: “absolutely it’s worth taking a discount for Sequoia, and Benchmark, and probably other tier-one VC firms.” Benchmark was a “third co-founder” from 2 to 400 people; Sequoia is “a machine” whose talent network sourced hires. His investing filter: talent, execution speed, and brand from being first — for a breakout team that’s proven its speed, “you just don’t look at the price and you just invest” — though AutoGPT, the fastest-growing GitHub project of all time, shows being first guarantees nothing. At the IPO prices Harry posed, Chase picks OpenAI at $1.5T over Anthropic at $2T.

Deep dive

1. Agents are hard because the black box sits inside a bigger black box

• LangChain builds developer tools: an open-source framework connecting models to tools, data, and computation, plus LangSmith for observability and evals. Chase’s core mechanism: models are “non-deterministic black boxes” taking infinite-dimensional natural-language input, and agents are not one model call but a series interacting with tools — “you get these cascading types of errors.” Two similar inputs can arrive at completely different answers; getting the system to behave reliably “is the hardest part.” Agents improved over three or four years for two reasons: better models, and better harnessing — LangChain works on the second.
• His answer to founders paralyzed by model progress: “What’s the alternative? You just don’t build anything?” The discipline is separating short-term hacks from long-term pieces — LangChain’s early JSON-DSL tool calling was a known hack. A few months later, OpenAI shipped native tool calling; Chase says they recognized it as the future and switched from JSON parsing. By contrast, “the models will never know your proprietary data,” so the system around the model endures. He recalls Cursor describing an operating principle as “erase the product” (or “delete the product,” as he thinks they called it): as models improve, you delete big parts of what you built.

2. Neutral harnesses exist because context must stay portable

• The case for a neutral harness is state: memory and context accumulate as the agent runs, and if that context is linked to a harness that’s linked to one model, “you’re kind of stuck.” Changing models is an offensive and defensive advantage — offensive to grab the best new model from anyone, defensive against lock-in and rate hikes.
• “Owning your intelligence,” in his framing, means an open, controllable agent across three pieces — model, harness, context. Model labs train models to be good at what’s in-distribution for them, “which is often coding”; once you’re doing sales, research, or law out of distribution, you want control over what the harness does.

3. Verification value flows to domain experts — and vertical agents

• If value comes from verification, “the people who win are the domain experts… whoever can write those verifiers, capture that domain knowledge.” Building evals is very hard, which is precisely why it matters — hence “I’m massively bullish on vertical agents,” including distinct vertical agent experiences per domain.
• The sequencing he’d give any vertical-agent founder: start at the context and harness layer, and only when you have users start post-training — “much higher ROI.” Would he post-train his own model if building a vertical app? “Not to start. Absolutely not to start” — though five years in, with heavy usage, “a realistic possibility.”
• On Mike Cannon-Brookes’ claim that enterprises “building their own models” is respectfully BS: Chase says much of it is post-training open-weight models, and defends it as a legitimate rung on a four-level ladder (control context → tweak harness → post-train open weights → train from scratch). Harvey and Ramp “have gotten all the juice they can out of the context and the harness” and arrived at post-training.
• Where post-training makes practical sense today: simpler sub-tasks inside a harness — classification, extraction, summarization, guardrails, routing. Whether the core loop-driving model should be post-trained or frontier is “an open question” — each new frontier release can get better at running the loop, though Harvey recently announced something in that vein.

4. Routing lives in the harness; the gateway case assumes coding or chat

• The technical distinction, prompted by Harry: harness-level routing lives in the code with more state awareness — prompt-cache warmth and per-model prompting strategies — while gateway routing is an API call that assumes a chat-completions interface. Factory and Cognition have written about this; Chase thinks routing “has to be harness aware… I don’t think you’ll have routing that works for law and works for finance and works for coding and have it all be the same.”
• His read on gateway incumbents: “most of OpenRouter’s usage, if I had to guess, comes from being a gateway, not a router” — users specify the model; the value is one API and one key. Smart routing is happening less than people think.
• When Harry asks “why am I stupid” for finding harness routing obvious, Chase steelmans the gateway: portability (you can’t modify Claude Code’s harness, so routing to DeepSeek may have to happen at the gateway — itself “another argument for open source harnesses”) and general domains such as coding or company-wide chat, where “‘Hi, how are you doing?’ doesn’t need Opus.” His two-to-three-year call: vertical agents route in the harness; horizontal chat and coding could route at the gateway.
• Is routing a commodity? “I don’t think routing is a commodity” — but building it is hard and its value is unstable, because “a new Opus or a new OpenAI model comes out, and it’s just better at everything. And so then the value of routing kind of goes down.”

5. Evals: trust yours, not theirs — and always look at the data

• Academic benchmarks are spent: “all the models that come out are benchmark-maxed against those evals, and so you don’t learn a lot.” Internal, domain-specific evals are useful and generally trustworthy — but must evolve, because models get better in ways your evals didn’t know how to capture.
• The practice he insists on: “just look at the data… don’t rely too much on a single number.” Traces only create value when insights flow back into context, harness, or model — post-training for the model, an agent run over traces for memory, and Engine, a LangSmith agent that runs over traces and tries to update harness code.

6. Memory is the unsolved layer — LangChain’s own failure proves it

• For an agent context layer, Chase expects something more like an agent around a database than a database alone; existing databases such as ClickHouse are good at storing and querying data in traditional ways. The hard part of memory is not storage — some companies store it in “a single file” — but distilling observations into context worth keeping. That meta-reflection is something “LLMs still struggle with a lot.” Chase says humans also find this hard: “I don’t know if we actually understand what goes on inside our brains.”
• The confession: LangChain launched then killed a commercial memory product. “We killed it because we didn’t find product-market fit… I think we actually tried to be too general.” Memory for Instinct or Muse looks different from an org harness, legal, or finance — so learning agents “at least to start, probably have to be domain-specific.” Still, he expects all products to have memory within a few years. When Harry pushes that this is obvious, Chase distinguishes the impact from the solution: “I don’t think anyone knows what the answer for memory is.”
• Continual learning, as he defines it — “a system that gets better over time as it runs” — doesn’t change the inner loop (LLM in a loop calling tools) but adds an outer loop: take traces, get learnings, fix context/harness/weights, measure, repeat. He says coding is getting there, and summarization and routing may be saturated, while meta-learning remains hard. Will models be meaningfully better at reflection in 12 months? “Probably. It seems like they’ll get better at everything.” He thinks progress could look much faster than before, perhaps because meta-learning matters to labs working on RSI, but says it’s tough to have an educated opinion without being inside a lab.

7. The agent-to-agent internet: every app grows a vertical agent

• Consumer will have both horizontal and vertical agents, with horizontal agents calling vertical ones. Two reasons apps should build their own: they learn things about you a general agent can’t (“how you interact with music on Spotify is very different than how you might interact with Strava”), and an agent wrapping a pile of APIs in one natural-language interface beats exposing raw APIs.
• Harry’s infrastructure worry — his concert-ticket agent checked every five minutes for three days, 1,000 times the human query volume — meets Chase’s answer: providers will expose their own agents (he cites Resy blocking a user’s account for frequent Instinct data pulls). Agents increase strain; systems need to handle burstier usage and scale more broadly, though he has no specific view on lower infrastructure layers. A provider’s vertical agent can interact with its underlying systems more efficiently than a third-party agent. On the lost-ads objection, Chase’s proposed response is ads inside the agent response: “I found these three things. You might also want to consider this other thing.”

8. Coding spend: 30% of salary, falling cost, rising tokens — and Anthropic hangs on it

• LangChain spends “roughly 30% of our engineering salary on coding tools” — a guesstimate — and spend has “gone down in the last 2 months” while tokens used have gone up. Chase’s three-step playbook: visibility (coding-agent usage tracked to LangSmith), basic budget controls, then controlling the harness. They built an in-house cloud coding agent, OpenSui — “the only way I code anymore,” driven from Slack — with prompt caching and routing.
• The cautionary specimen: a plugin snapshotting the DOM and sending it back every agent turn burned “$40K in a week on that one thing” — it wasn’t malicious or intended; it was a mistake. Some big spenders are the senior engineers he trusts 100%; other spikes come from misconfiguration.
• Frontier models are getting relatively cheaper — he cites Astra’s price-performance compared with things that existed a year ago — but agents and multi-agent systems use more tokens, and coding costs have risen. He says providers are making cost controls easier: tools can now be added dynamically mid-flight without busting the prompt cache, which previously could cause a large cost increase.
• Harry’s thesis: this percentage is the fundamental determinant of Anthropic’s enterprise value — at 5% the $2T IPO is overpriced, at 300% dramatically underpriced. Chase agrees it underpins the value but says organizational design makes the future hard to predict — perhaps one person drives 100 agents at 3× the cost of human salaries, or perhaps spend stays flat because those rarer engineers get paid more. Software engineering is the only domain where agents have really had an effect so far; he expects agent impact in other domains to be much greater than today.
• From LangChain’s annual developer surveys: accuracy has been the number-one concern for two years running, “basically double the next 2” (latency and cost). Recently, cost has risen as a concern for large enterprises; security is a higher priority there than in the overall survey, but still below accuracy and cost.

9. Open models are narrower than hyped; inference demand is the great underestimate

• Enterprise fear of open-source Chinese models is “largely unfounded” — data leaking back to Chinese labs is “just ridiculous” if you’re hosting on Fireworks or yourself, and concerns about model values are where evals help, though there may be nuance in some domains. Comparable U.S. open-source models within 12 months? “Absolutely” — citing NVIDIA’s ecosystem push and Reflection’s recent launch, still a little behind some Chinese models on some benchmarks.
• What his vantage point shows that the market doesn’t: open models get lots of usage but “for narrower tasks than people think” — routing, guardrails, extraction, summarization — and “I don’t think they’re really driving the core agentic loops.” He sees good ROI in post-training models for simple, high-volume use cases.
• His two biggest underestimates since founding: demand for inference and demand for coding agents. He invokes Eric Vishria of Benchmark, who said, “Everything just works. This market is so big,” and separately recalls Thomas Tunks saying the inference market is big. Applying that broader point to inference, Chase says: “Fireworks will work. AWS will work. Crusoe will work. There’s just so much demand for it.” In the quick-fire, his biggest change of mind lands here: LangChain debated building a gateway a year and a half ago and passed — “that was 100% the wrong decision.”

10. Where he’d invest, why being first matters, and taking the tier-1 discount

• Playing Harry’s newest partner, his portfolio: inference; low-level infrastructure primitives (Modal, Temporal, Supabase, Convex — “there’s going to be way more software written”); and vertical agents, with no overweight vertical. Echoing Patrick Collison, the opportunity is unsexy areas with low AI penetration and deep domain expertise capturable in data or harness logic. Three prerequisites across winners: elite talent (“Lynn and Fireworks… they wrote PyTorch”), execution speed, and brand from catching “lightning in a bottle.” LangChain saw 100 frameworks emerge in the two months after its launch, but Chase says brand and execution helped it ride its early virality. For a breakout team with a strong technical roster that has proven it can execute quickly, “you just don’t look at the price and you just invest.”
• Being first is “absolutely” more important than ever — “Anthropic and Claude Code… that was a huge advantage” — but execution must follow: AutoGPT, “the fastest-growing GitHub project of all time,” shows that being first doesn’t guarantee success. His bull case for the $9B Jev round is a new model category associated with the company and a talented team that can execute; he says he doesn’t know the team super well, so that talent would be an assumption at that valuation.
• On tier-one money: Benchmark was a “third co-founder” from two people to about 400, while Sequoia (Sonja) “help[s] with everything… they’re just a machine,” including strong candidates from its talent network. Chase says it’s absolutely worth taking a discount for Sequoia and Benchmark, and probably other tier-one firms.
• Harry rails against fundraising posturing; Chase says he doesn’t like fundraising and valued the relationship he built with Tom at IVP over more than two and a half years before the Series B. Quick-fire residue: SmithDB, LangChain’s trace database, was the hardest technical build; Chase says the company has “500 different companies” it is “frenemies with.” He picks OpenAI over Anthropic at the IPO prices Harry posed because OpenAI has “more diverse paths” to large outcomes. He wishes he’d known what today’s harnesses looked like when he started in 2022, and his five-year excitement is creative AI like Suno, not only enterprise efficiency.