Write, Change, Recall, Forget: MongoDB's Pete Johnson on How Retrieval Drives Agent Performance

Summary

• Johnson’s central framing is that agent performance increasingly depends on speed, scale, and retrieval quality; token-maxing creates both cost and relevance problems. He cites Uber “burning through their token budget for 2026 in 13 weeks,” notes academic findings that only “the first 7K and the last 7K tokens” in a context window matter most, and argues the goal is “just the right 200,000” tokens per agentic loop rather than a million every pass — pushing enterprises toward selecting the right context instead of maxing the window.
• Johnson insists embedding models are not commoditized, contrary to market consensus — the thesis behind MongoDB’s Voyage AI acquisition. Voyage models are typically at the top of Hugging Face’s MTEB benchmark, and he cites “as much as a 14% improvement” over some other embedding models, potentially “the difference between a hallucination and a correct answer”; rerankers add another 5–10%. His kicker: “Anthropic does not have an embedding model on the market. They recommend us.”
• MongoDB’s stated operating position is about $2.5B in fiscal-2026 revenue, roughly 2–3% of a $100–110B database market, and presence in 75% of the Fortune 500, with AI expanding the set of problems where speed, scale, and retrieval quality favor the document model over SQL. Nathan cited roughly $30B in market cap and framed the Voyage acquisition at $220M—under 1% of that figure—as a test of whether incumbents can absorb model-layer value cheaply; Johnson’s answer is that MongoDB’s existing sharding, replication, and security made vector search easier to add, while Voyage was differentiated.
• Agent memory is the current frontier and “the hardest problem with agents right now” — captured in a colleague’s formula Johnson quotes: “Write, change, recall, forget.” Forgetting is the hardest part; sophisticated systems now use token budgets, RBAC-shared memories, and a new “taxonomic memory” type that retrieves only the five relevant vertical-specific terms per loop instead of cramming in all hundred. Johnson’s broader warning is: “We’ve been building databases for 60 years, and we’ve been building agents for about 18 months. Nobody knows all the answers yet.” There is “no LAMP stack for agents yet.”
• Enterprise deployment reality check: most Fortune 500 companies Johnson talks to are doing employee-facing use cases with human-in-the-loop, not fully autonomous customer-facing AI, because the risk-reward is asymmetric. A leaked salary between coworkers is bad; leaked customer data is a scenario where “a VP has to take the blame.” ROI success comes down to problem selection — pick problems where metrics already exist (call centers, SDLC), because “if you don’t already have metrics for how something is performing, you won’t know if it got better.”
• His most contrarian observation: the two most sophisticated AI customers he met across seven countries in 2026 were in Mexico City and São Paulo, not the US. Unlike the cloud and mobile waves, where missing hyperscaler data centers or radio towers created geographic moats, “access is democratized now” — models, vector databases, embedders, and rerankers are far more widely available, eroding the assumed American lead.
• A vector-DB startup founder’s counter to incumbency, which Johnson says has “some truth”: most data entering vector databases “has never been in a database before at all” — PDFs in SharePoint, audio, video and other unstructured data — meaning the AI wave is expanding the indexed-data universe rather than just redistributing existing database share.

Deep dive

1. Database design is downstream of whatever resource is scarce — and the scarce resource keeps changing

• Johnson’s origin story, told with a personal hook: E. F. Codd’s IBM white paper birthing SQL dates to June 1970 — “I happen to have been born in February of 1970, so I am as old as SQL.” In 1970 storage was by far the most expensive resource, so normalization — never storing a piece of data twice, spreading his and his wife’s shared address across three tables — was rational design.
• By MongoDB’s first commit in October 2007, “47 years of Moore’s law” had made time the scarce resource: cloud, mobile, no weekend downtime. Denormalize the address into two JSON documents and “what you get in return is one disk read” instead of three — part of the reason for MongoDB’s speed, compounded by keeping JSON (BSON on disk) in the same format from disk to client.
• He goes further than most vendors would: “our education system has a bias, a historical bias that thou shalt always normalize” — valid for many use cases, but AI is expanding the set where it isn’t.

2. Positioning: 2–3% of a $100–110B market, and AI widens the wedge

• The numbers as stated: Nathan cited roughly $30B in market cap; Johnson said MongoDB had about $2.5B in fiscal-2026 revenue, roughly 2–3% of the $100–110B database market, and presence in about 75% of the Fortune 500. Enterprises split between workload-by-workload database choices and all-in strategies, driven by greenfield-vs-brownfield mix and where talent already sits.
• The through-line Johnson returns to all episode: MongoDB wins on “speed, scale, and retrieval quality,” and the third has become increasingly important for agentic architectures as companies strategize around token maxing and evolve new architectures later in 2026. At a North American Big Three automaker, Johnson says MongoDB trained about 250 engineers for three days, spanning dealer efficiency, mechanic tooling, and in-car experience.

3. Correction of record: MongoDB is schema-flexible, not schemaless — and that’s why vector search was easy to bolt on

• Johnson’s pushback on Nathan’s framing: documents in a collection can have different shapes, but that’s flexibility, not absence of schema — and “anybody who’s ever had to change a SQL schema that’s already in production… knows the pain of what I’m talking about.”
• The vector story started unusually, with lexical search: in 2020 MongoDB noticed customers standing up their own Apache Lucene servers next to clusters for keyword retrieval, so it shipped Atlas Search. Then the leap: “at the end of the day, a vector is an array of floats” — just another attribute on a flexible document, index it and you have vector search. That enabled hybrid search with pre-filtering: lexical on the title, vector on the synopsis, filtered to books published after 2000 — “three levers of query power.”
• The 2025 Voyage acquisition completed the arc: use Voyage on any vector DB, use any embedding model on MongoDB, or take the “better together story.”

4. Bitter-lesson engineering, vendor edition: keep deleting the developer’s plumbing

• Rank fusion and score fusion, newly announced aggregation-pipeline stages, collapse what used to be separate vector and lexical calls plus hand-rolled merging into “one round trip call to our servers,” returning a pre-sorted result set by rank or score. Johnson’s sales joke: “There are 2 kinds of developers: those who love MongoDB and those who haven’t tried us.”
• Same pattern twice more: $rerank, released in the spring, folds the vector-search-then-reranker round trip into one API call, and auto embeddings watches a designated attribute — any change or new document triggers re-embedding through your chosen Voyage model and index update automatically, removing the need to maintain a homegrown embedding pipeline. The stated philosophy: “more developers will build AI agents in the next three years than did in the last three years… we have to lower the learning curve.”

5. The chunking trade-off, and Voyage’s two tricks for dissolving it

• Johnson lays out the classic iteration loop: sentence-level chunks lose context, three-page chunks raise storage cost and drown the sentence you wanted — “the answer is always it depends,” so developers run three or four test cycles. Voyage’s contextualized chunking (v3 last summer, v4 released in the last six weeks) flips the script: send the target sentence and its surrounding context as two strings, get back one array of floats balancing both “in some sort of superposition” (Nathan’s phrase, which Johnson confirms: “that’s exactly how it works”) — yielding “better retrieval quality with a smaller chunk size, which otherwise is not possible.”
• The second trick is Matryoshka embeddings, named for Russian nesting dolls: embedding dimensions, often ranging from 256 to 2,048, are ordered, so testing 512 dimensions after generating 1,024 means “you just lop off the last 512” instead of re-embedding the entire corpus. Neither feature solves the quality-vs-cost decision, but both compress the iteration loop.

6. When does any of this matter? Johnson’s thresholds — and the anti-commoditization claim

• Nathan’s honest self-audit as a “business of one”: a roughly 1GB personal retrieval corpus over five years of email, Slack, DMs and transcripts, vibed together with a Gemini embedding model and no evals — how would he know what he’s missing? Johnson doesn’t flinch: “you’re not going to offend me… most people start with Postgres and pgvector” plus whatever embedding model their cloud defaults to.
• The thresholds as stated: milliseconds matter when they matter to your use case; scale means “in the neighborhood of 100,000 vectors”; and on quality, “most people think embedding models are commoditized, and that is not true” — Voyage models are typically at the top of Hugging Face’s MTEB benchmark, with as much as a 14% improvement compared with some other embedding models. Rerankers can add another 5–10%. The mic-drop: Anthropic has no embedding model on the market and recommends Voyage.

7. From simple RAG to token-maxing hangover: the case for sophisticated memory

• Johnson’s compressed history: ChatGPT’s 2022 architecture was query-in, answer-out; RAG provided a way to inject proprietary data; tools and MCP helped overcome the knowledge-cutoff problem in 2025; agentic looping then exposed that “the context window is still very dumb” — fresh every pass — birthing crude memory: cram the session in, then cram three days of sessions in.
• Two costs of that crudeness: dollars (Uber’s 2026 token budget gone in 13 weeks, which “grabbed a lot of headlines”) and function — academic studies showing the first and last roughly 7K tokens are most important while “the stuff that’s in between can end up just muddying and confusing the LLM.” The new goal: “what if I could choose just the right 200,000 for this agentic loop?”
• The emerging Fortune 500 pattern he highlights: taxonomic memory — of a hundred vertical- or company-specific terms, retrieve only the five relevant to this loop, via vector/lexical/pre-filter combinations, chasing “some perfect context window for this pass” that you never quite reach.

8. “Write, change, recall, forget” — and why forgetting is the hardest part

• Nathan’s confession of memory-system pain — monthly log summaries rolled into yearly summaries under a Karpathy-inspired wiki, with models keeping abandoned projects “as an open thread for months and months… dude, I never actually did that” — draws Johnson’s frankest admission: “This is the hardest problem with agents right now… We’ve been building databases for 60 years, and we’ve been building agents for about 18 months. Nobody knows all the answers yet.”
• The pattern he sees maturing: the application author has two responsibilities — query the memory system with a token budget (“give me the best 50K tokens”), then send the answer back so the system curates and writes it, with RBAC so colleagues in the same job type share memories. The framework he quotes verbatim from a coworker: “Write, change, recall, forget” — memories have a half-life.
• On graph structures: customers commonly go a half-dozen to a dozen layers of graph for larger, more heavily categorized corpora, then do vector retrieval inside a leaf node — all in one JSON-based platform. He worries about memory architectures relying on multiple passes of the LLM to shrink the corpus rather than lower-cost embedders and rerankers, because tokens and latency add up. An example he offers: ElevenLabs, MongoDB’s biggest customer, running micro-agents per customer atop its audio platform.

9. Build vs buy: the winners picked better problems, not better tech

• From roughly 100 customer conversations across seven countries, three camps: “I bought a license for this one tool and I’m done”; PoC purgatory with weak ROI; and the sophisticated minority optimizing memory systems. On the middle group: “The biggest difference isn’t in how they’re applying the tech. It’s what problems they chose to try to solve.”
• His problem-selection filter: of your top 10–15 business problems, which have good data and existing metrics? “Bad data quality and bad security posture don’t get solved by AI, they get amplified by AI.” Call centers are popular because cost-per-call and volume are already measured; in software delivery, “lines of code is a terrible metric” — idea-to-production speed is what matters.
• The repeated caveat that frames everything: “There’s no React and Angular. There’s no LAMP stack for agents right now” — enterprises should expect meaningful customization even when they buy.

10. Deployment reality: employee-facing, human-in-the-loop — because risk is asymmetric

• Nathan’s field report from what he believes is a one-location Greg’s Pizza in Detroit — an AI answered the phone, handled the happy path impressively, but misread his adversarial “what company powers this?” as a corporate pizza order — prompts Johnson’s sobering counterweight: most Fortune 500 companies he talks to are doing employee-facing use cases with humans in the loop.
• The reasoning is both ROI (KPIs and bonuses already exist for employees, so AI-driven jumps are attributable) and security: a salary leaking between coworkers is bad, but customer-to-customer data leakage is a scenario where “a VP has to take the blame.”

11. The Voyage deal as a macro tell — and the data that was never in a database

• Nathan’s pointed question: MongoDB implemented vector search in its own platform but bought a model startup for a price he cited as $220M — under 1% of the roughly $30B market cap he cited — so is the lesson “infrastructure is the big winner, models get commoditized, incumbents defend against startups”? Johnson’s answer stays operational: because of the JSON base, vector search was “relatively straightforward” to add atop a decade of Atlas sharding, replication, and security “that some of the net new vector database companies… have to catch up with and struggle with,” while Voyage was genuinely differentiated where the market wrongly saw commodity.
• The startup founder’s rebuttal Nathan relays — “most of the data coming into our vector database has never been in a database before at all,” sitting in data lakes and unstructured piles — Johnson says “there’s some truth to that,” citing video, audio, and SharePoint PDFs now becoming searchable through embeddings.

12. The world isn’t behind — and the cycle time keeps shrinking

• The surprise Johnson volunteers: after spending time in Amsterdam and London and touring Toronto, Bengaluru, Mexico City and São Paulo, “the two most sophisticated customers I talked to this year were in Mexico City and São Paulo” — companies that assumed US competitors were ahead when the opposite was true. His explanation isn’t US conservatism but democratized access: unlike cloud (“if one of the hyperscalers didn’t have a data center in your country yet, you were kind of out of luck”) or mobile (“an iPhone’s not going to do you much good” without towers), the infrastructure is now widely available.
• Closing technical nugget: shared embedding spaces, released in January — four versions of Voyage 4, the text model, including an open-weight, free “nano” on Hugging Face, share one embedding space, so you can embed your corpus with the large model and run development queries through nano on a laptop, potentially eliminating token costs during development apart from embedding the corpus, at a retrieval-quality hit.
• His final framing of the pace: Anthropic published MCP “on the Monday before Thanksgiving 2 years ago and by March, all of its competitors had embraced it”; token maxing surfaced in early March “and by April it was over as a conversation piece.” The cycles are faster than any prior wave — “there’s still a lot for all of us collectively to learn.”