The Rise and Fall of the Vector DB category: Jo Kristian Bergum (ex-Chief Scientist, Vespa)
Summary
- The guest’s central call is that vector-database companies may survive, but the standalone category will not. Vector search now exists in PostgreSQL through pgvector and in Elasticsearch, Solr, Vespa, and other systems; the durable abstraction is search, while vectors become an implementation detail: “I’m not saying that the companies are dying, but I’m saying that the category is dying.”
- The category attracted more venture capital than its likely market structure could support. The host estimates roughly $230 million went into vector databases—more than MongoDB raised across its entire lifespan—and cites unverified reports that Pinecone reached “like $100 million ARR” rapidly before losing momentum: “They cannot all win.”
- The practical RAG stack begins with clean data and a 30-year-old keyword baseline, not a specialist database. The guest recommends BM25 first, then an off-the-shelf embedding model for hybrid search, and finally reranking only when its latency and cost are affordable; PostgreSQL may suffice at reasonable scale, but search-critical businesses should consider a dedicated retrieval engine.
- Long context eliminates some legacy RAG workloads without eliminating retrieval itself. A single PDF or roughly 300 articles might fit directly into a Gemini context window, especially without high-QPS requirements, making an early-2023 vector pipeline unnecessary. But 170,000 documents already represent 36 million tokens: “You’re not going to load all of that…for a single query.”
- Embeddings remain foundational, but cosine similarity alone cannot produce strong search. Freshness, authority, metadata, and reranking still matter, while large recommendation systems use embeddings to retrieve an initial candidate pool before cascading toward perhaps 100 results: “Embeddings are here to stay.”
- The guest sees PostgreSQL as reasonable at moderate scale but rejects pushing model inference into the database. He is “not bullish” on giant SQL pipelines that transform, embed, and rewrite data because inference and storage have different scaling properties—and because he wants explicit control over cost and performance.
- The next embedding opportunity may be domain-specific, multimodal document retrieval, though the economics remain uncertain. The discussion points to Voyage as a leader in domain-specific models and notes its acquisition by Nvidia; Jina AI is doing strong work, especially in European languages. The guest wants models for legal, finance, and health documents that can embed a page screenshot without OCR, but API compute and batching make it “a difficult business model.”
Deep dive
1. Vector databases became a feature before becoming a durable category
The guest traces the category to November 2022, when an OpenAI Cookbook example connected ChatGPT to users’ data through embeddings. He helped write its Chroma example and was an angel investor, but says developers absorbed an “unnatural connection” that retrieval in RAG had to mean vectors.
Pinecone then framed embeddings as a new infrastructure category: if every AI application needed embeddings, every application needed a vector database. The host cites rumors—not verified figures—that Pinecone raced to “like $100 million ARR,” while the guest sees its newer developer-focused positioning as a return to its roots.
Competition from Turbopuffer matters, but convergence matters more: pgvector, Elasticsearch, Solr, Vespa, and numerous databases all offer vector search. The guest’s distinction is categorical: “I’m not saying that the companies are dying.”
The host’s venture-market check: approximately $230 million entered vector-database startups, more than MongoDB raised over its lifetime. MongoDB established a broader NoSQL category; vector databases are “too narrow” to stick the same way.
2. Search quality earns complexity one layer at a time
For workloads already living in PostgreSQL at reasonable scale, the guest considers pgvector fair: it added IVFFlat, HNSW, half-precision, and binary vectors. If search quality determines the business, however, a dedicated retrieval engine deserves consideration.
The RAG sequence starts by inspecting and cleaning the data, especially PDFs. BM25—“keyword matching” that has existed for 30 years—provides a strong baseline; hybrid embedding search follows, then reranking if the application can afford its latency and cost.
The host notes that reranking may add only “3% to 4%” and asks how the stages should be sequenced. The guest resists a universal recipe; at large scale, recommendation systems cascade from embedding retrieval through several reranking layers until perhaps 100 candidates remain.
3. Model inference and databases have different scaling laws
At thousands of QPS, the guest would avoid making a remote embedding API that returns JSON floats a dependency and would prefer something local and faster. He recalls treating a roughly 300-millisecond endpoint call for a large float as acceptable, but is shifting toward the view that API services are easier to build with for smaller workloads.
He remains “not bullish” on PostgresML-style designs that push embedding and LLM inference into huge SQL statements. Storage and inference scale differently, and opaque execution weakens control over cost and performance; the host concedes that the database-versus-external-system boundary is a recurring tension.
4. Long context kills obsolete RAG demos, not retrieval
The guest says readers wrongly converted “the vector database category is dead” into “RAG is dead.” Augmenting AI through retrieval or search remains relevant “for a very long time,” even if the implementation is no longer a dedicated vector store.
The nuance is workload-specific: one visual PDF or 300 articles might fit directly inside a Gemini model, particularly when high-QPS requirements are absent. Context expanded from 4K or 8K to 10 million while people kept reproducing early-January-2023 demos designed around the old limits.
The guest’s cryptic tweet says Llama 4 will reignite the long-context-versus-RAG debate and resolve it “not in the way that you want.” The host calls blanket “long context kills RAG” claims engagement farming.
The concrete boundary is 170,000 documents totaling 36 million tokens—already too much to load for every query.
5. Better data creation unlocks graphs and document-native embeddings
The guest’s GraphRAG objection is not traversal: graph databases handle edge traversal, random access, and jumping across edges well. The real bottleneck is constructing entities and relationships, and he “hate[s] that connection” where adopting a concept supposedly requires one specific database technology.
GraphRAG might beat Vector RAG in some cases, or work as a hybrid. The discussion notes that LLMs could ease the previously hard task of generating triplets, while the possibility that knowledge graphs may no longer be a “dirty word” is left hedged: “Maybe, maybe, maybe.”
The guest’s desired frontier is domain-specific embedding models for legal, finance, and health documents, ideally using vision-language backbones to embed screenshots without OCR. The discussion points to Voyage as leading this direction and notes its acquisition by Nvidia; Jina AI is doing “a lot of great things,” especially in European languages. Yet vendors must fund inference and batching through an API, pushing companies toward enterprise search and other higher-value layers.