World Models & General Intuition: Khosla's largest bet since LLMs & OpenAI
Summary
General Intuition’s core asset is Medal’s 3.8 billion game clips—a “Common Crawl for interactivity” pairing visual states with semantic actions and unusually dense examples of peak human behavior. Its roughly 12 million users retroactively save memorable moments rather than entire sessions, yielding both exceptional play and adverse events. The moat is not video volume alone: Medal built the recorder, public sharing graph, privacy-preserving action labels, and distribution together.
GI’s current vision-only policy appears capable before reinforcement learning: “pure imitation learning,” with no fine-tuning or game-state access. It processes pixels, predicts actions in real time, plays against humans, and uses four seconds of memory to recover from mistakes. When asked whether it was given a goal, Pim answered yes; the discussion did not specify how that goal is encoded. The policy inherits exceptional maneuvers from highlight data. As the host clarified, this is not a “Move 37” discovered through RL; it is replication of superhuman examples already present in a dataset whose baseline is “peak human performance.”
Pim defines a world model as an action-conditioned state generator, not a video model predicting the likeliest or most entertaining next frame. From one second of context, GI showed roughly 20-second generations preserving location through smoke, rapid camera movement, zoom changes, and temporary occlusion. A generated explosion even inherited real-world camera shake absent from the underlying game—evidence, in Pim’s framing, that physical-video pre-training might transfer beyond games.
The immediate commercial wedge is an API replacing hand-coded player controllers and behavior trees inside major games and game engines. Better skill-graded bots can support player liquidity at 3:00 a.m.; the same “frames in, actions out” foundation might extend into simulation, manufacturing, and robotics when machines accept game-controller-like inputs. GI does not claim zero-shot deployment: Pim’s bet is that customers might need only 1% or 10% of today’s domain data because pre-training becomes post-training.
The episode’s financing story reflects both the perceived scarcity of the dataset and the difficulty of valuing proprietary data before modeling it. The host said OpenAI offered $500 million for Medal’s data and presented GI as having turned it down; the seed round was described during the episode as both $133 million and $134 million, led by Khosla Ventures as Vinod Khosla’s largest such bet since OpenAI. Pim’s advice to other data owners: build models first, identify capabilities that are not already inevitable, understand the buyer’s intended use, and seek equity where possible because synthetic data could rapidly reduce licensing value.
Pim expects language models to remain valuable orchestrators rather than treating them as simply obsolete. In the discussion, the host argued that text compresses three-dimensional experience and that spatial models may provide a better generalization backbone, while autoregressive token generation poorly matches continuously changing environments. Pim’s architecture resembles SIMA 2’s “puppet master”: a language or vision-language model steering many embodied policies, with text and speech becoming actions rather than the whole substrate.
GI’s compounding loop is to turn Medal’s billions of clips from imitation examples into playable environments for RL. Pim calls the archive “the episodic memory of humanity in simulation”: perhaps two or three memorable minutes selected from three hours of play, including precisely labeled crashes and other negative events. The 2030 aspiration—80% of AI-driven “atoms-to-atoms” interactions and “100x more in simulation”—is an extreme market-share ambition, with simulation expected to lead because it has fewer physical and safety constraints.
Deep dive
1. The policy already behaves like a gamer without reinforcement learning
In a four-month-old demo, the vision-only agent received frames and predicted actions, even compulsively checking the scoreboard like human players. Its four-second memory let it recover after getting stuck, though its early play still contained visibly awkward decisions.
The current base model ran entirely in real time against human players with “no RL, no fine-tuning” and no access to game state. When asked whether it received a goal, Pim answered yes; he also described the policy as purely pixel-based, predicting actions from frames rather than game states.
The revealing sequence combined failure and excellence: the agent made a mistake Pim said a human would never make, unstuck itself, then executed a distant maneuver the host regarded as superhuman. Pim’s explanation was inheritance, not discovery: “The baseline of our data set is peak human performance.”
The host sharpened the distinction: this was not a “Move 37” produced through reinforcement learning, but imitation of extraordinary human examples. That also bounds the result—the model can reproduce capabilities represented in Medal’s highlights before GI has built the RL loop intended to exceed them.
2. Action-conditioned generation preserves a world through smoke and motion
GI both pre-trains world models from scratch and fine-tunes open-source video models for physical transfer. Pim emphasized mouse sensitivity and rapid movements that gamers expect, then showed holdout generations receiving only one second of video context plus recorded actions; everything afterward, for roughly 20 seconds, was generated.
One explosion caused the generated camera to shake even though the original game would not. Pim interpreted this as the model importing behavior from physical-world video, and argued that combining such transfer with GI’s action-prediction models could push training “way beyond games.”
Partial observability was the stronger spatial test: after smoke obscured the view, the generation emerged in the same location; while sniping, it retained object placement through zooming and temporary loss of view. Smaller distilled policies made more mistakes, such as running into a wall, but still operated in real time—making parameter count, and potentially serving cost, tunable.
3. Medal’s recorder created a privacy-preserving behavioral dataset
The episode’s headline numbers were 3.8 billion clips and about 12 million Medal users; the host contrasted that with Twitch’s roughly 7 million monthly active streamers. Pim called the resulting collection perhaps one or two orders of magnitude larger than any other ground-truth, action-labeled video dataset online.
Medal won by perfecting capture before trying to bootstrap consumption. Its recorder always runs in memory and, when the user presses a button, exports only the preceding sequence to disk and phone—removing start/stop friction and capturing natural behavior. Fortnite, COVID-era growth, and Discord’s “connective tissue” subsequently helped the social network accelerate.
Privacy shaped the labels: Medal chose not to log individual keyboard inputs such as W, A, S, or D. Over roughly 18 months, thousands of people mapped each game’s controls into semantic actions; those actions can later be converted into generic computer inputs, but not reconstructed for an individual user.
The host’s pushback remains important: much of games—and life—is boring, while clips deliberately select unusual moments. The host also observed that behavior is materially shaped by context-window length; Pim agreed, but did not elaborate on that point.
4. Research papers turned a gaming company into an independent lab
SIMA’s reported transfer result supplied Pim’s first strong clue: train navigation on nine games, hold out the tenth, and the general agent performed roughly as well on that tenth game as an agent specialized for it. Medal possessed precisely the cross-environment action data needed to test that scaling thesis.
DIAMOND provided the feasibility proof. Pim recalled a playable world model running around 10 FPS on a consumer 4090 using roughly 90 hours of data—he variously remembered figures around 87–95 hours—then cold-emailed the student team. DIAMOND contributors, GAIA-2 research lead Anthony Hu, and additional researchers ultimately joined GI.
The team initially believed Medal’s data was chiefly useful for Genie-like world models. They changed their view after discovering how far pure imitation learning could scale in parallel, reserving playable world models for the later RL phase: “We essentially have sort of the internet, or like Common Crawl, if you will,” but for interactivity.
Pim described himself as a self-taught infrastructure engineer who knew CUDA, transcoding, and video systems but found modeling foreign. After completing François Fleuret’s roughly 30–40-lecture deep-learning course, he felt like “the guy who just got his driver’s license,” while his research co-founders remained the “F1 drivers.”
5. World models attack interactions that conventional simulation cannot afford
Pim’s definition is explicitly causal: a video model might predict a likely or entertaining frame, whereas a world model must represent possible outcomes and generate the next state from both the current state and an action. It is “a world that is accurately generated based on the actions that you take.”
His attempt to build a rudimentary differentiable physics engine in PyTorch exposed three sources of simulation complexity: the number of agents, each agent’s degrees of freedom, and how much information each action reveals. Saying “water” rather than “fire” can redirect human behavior enough to explode the outcome space.
That combinatorial burden motivates GI’s “maximal bet on video transfer” for interactions difficult to hand-simulate, not a claim that conventional engines disappear. Pim still expects traditional simulation to retain a large market wherever representative video is scarce.
Games also preserve information that ordinary video loses. A YouTube pipeline may require pose estimation, inverse dynamics, and an estimate of where the person was looking; in a game, hand-controlled camera motion already captures those “optical dynamics.” GI will begin with controller-sized action spaces, then may learn action embeddings inside a more general space.
6. Spatial models and language models are likely to converge
SIMA 2 impressed Pim less as a policy breakthrough than as an orchestration blueprint: text, drawings, and markings could steer action sequences, while Gemini could become an independent “puppet master” coordinating many SIMA instances. He expects GI’s initially non-text-steerable policies to move toward a similar architecture.
On Fei-Fei Li’s splat-based approach, Pim saw value in a verifiable underlying three-dimensional representation reusable by game engines. His reservation was that, as he understood it, the approach was not yet interactive—“the whole point” of a world model in his definition—and that it involved a much larger output space with uncertain scaling. He guessed the team would eventually reach the interactive piece but said it was too early to judge.
Pim said Yann LeCun’s past argument that LLMs were a dead end helped inspire him, but the transcript does not establish that Pim himself adopted that categorical conclusion. The host separately argued that humans use text to compress three-dimensional experience, that text enabled science and literature, and that autoregressive token generation poorly matches a continuously changing environment.
Pim said GI could import language-model research and earlier predicted that an orchestrator VLM would manage its policies. In his framing, text and speech can become actions within a broader system; he did not present a settled answer about whether language-first and world-model-first labs will converge.
7. Games provide the first revenue wedge and simulation the expansion path
GI is already working with large game developers and game engines to replace player controllers or deterministic behavior trees. The proposed Anthropic-like API is simple: customers stream frames and GI predicts actions; engagements may also include custom policies or distilled models, but Pim was categorical that “we don’t sell data.”
Better bots have direct retention value when player liquidity is low, such as at 3:00 a.m. Pim conceded that users generally do not want an all-bot match; the target is a convincing mixture of humans and skill-graded bots, which Medal can calibrate because it knows players’ demonstrated ability.
The host pointed to GTA V role-play, Truck Simulator, and PowerWash Simulator as behavior-rich examples. Pim agreed that these environments contain purposeful, real-world-like behavior and emphasized how seriously some gamers reproduce truck-driving setups; he said Medal has more people using steering wheels in driving simulations at a given moment than Waymo has cars on the road.
Robotics and manufacturing customers are already involved, but the transfer condition is restrictive: the robot must accept gaming-like controls. GI is not promising zero-shot autonomy; Pim hopes its pre-training lets another company collect perhaps 1% or 10% as much domain data and shift today’s hardest pre-training work into post-training.
8. The data deal became a test of whether to license or build
The host introduced GI as having rejected a $500 million OpenAI offer for Medal’s data, then raised a seed quoted as both $133 million and $134 million. Khosla Ventures led what the episode described as Vinod Khosla’s largest single seed bet since OpenAI.
Pim described Khosla’s diligence as a first-principles backcast: draw the company in 2030, work backward to today, and defend every technical and commercial assumption under questioning. That process tests whether a grand vision is connected to something real; once convinced, Khosla “backs fully” and opens doors.
Pim’s conclusion on data valuation was blunt: “I don’t think you can value it unless you actually model it yourself.” Owners should test whether their data unlocks non-obvious capabilities, determine whether it belongs in LLMs, world models, or another system, and understand the buyer’s intended products—especially where licensing might “bite the hand that feeds us.”
He also warned that better labeling models and synthetic generation can rapidly reduce the amount of ground-truth data required. Where possible, he would seek substantial equity rather than a one-off license; where a lab refuses to explain intended use, his general inclination is not to proceed.
9. Playable memories are the bridge from imitation to RL and robotics
Pim calls Medal “the episodic memory of humanity in simulation”: a user may compress three hours of play into two or three minutes of the most memorable, shareable, or out-of-distribution events. That archive includes people “at their best and worst,” with titles and action histories identifying crashes and other adverse outcomes.
GI’s RL plan is to make every one of those billions of clips playable inside a world model. A known sequence leading to a Euro Truck Simulator crash can become an environment in which reward models score performance around negative events—turning self-selected failure data into repeated interactive practice.
The same capability creates a consumer product: users or friends could enter and replay a saved clip rather than merely watch it. Medal already serves millions and millions of video views a day, giving GI a path to world-model entertainment and fast product integration without inventing a separate consumer network.
The name General Intuition comes from the AlphaFold-era idea of mimicking gamers’ intuition, even when those gamers were only amateur biologists. Pim’s north star is eventually to represent scientific problems in three dimensions and combine spatiotemporal perception with language reasoning to work directly on those problems.
10. The 2030 vision puts simulation far ahead of the physical rollout
GI argues that its data moat permits more open publication because competitors cannot readily reproduce the training set. Pim cited a partnership with Kyutai, a Paris open-science lab, and invited universities to pursue projects such as negative-event prediction from truck data involving crashes in India.
Scaling ambitions progress from thousands to tens of thousands and eventually hundreds of thousands of GPUs. Pim framed the absence of severe data constraints as a research advantage: the team receives weekly results it had not expected and can run experiments unavailable to institutions working without action-labeled interactive video.
For 2030, Pim wants GI to become “the gold standard of intelligence,” reasoning that any sufficiently long sequence is fundamentally causal and therefore requires spatiotemporal reasoning. He divided AI’s progression into “bits to bits, atoms to bits, bits to atoms, and atoms to atoms,” crediting Andrej Karpathy for the framing.
His concrete target is responsibility for 80% of AI-driven “atoms-to-atoms” interactions, with specialized robotics stacks taking the other 20%, and “maybe 100x more in simulation.” Simulation should lead because it faces fewer safety and physical constraints; Pim expects scientific and virtual environments to provide the initial takeoff before intelligence becomes the bottleneck-clearing layer for robotics.