After LLMs: Spatial Intelligence and World Models — Fei-Fei Li & Justin Johnson, World Labs
Summary
World Labs is betting that spatial intelligence will complement language as the next foundation-model frontier, with compute finally large enough to attack it. Justin Johnson estimates roughly 1,000x more performance per card since AlexNet and training across hundreds to tens of thousands of GPUs, producing “a millionfold more” compute per model. Visual, spatial, and world data require far more processing, making world models a plausible next scaling frontier.
Marble is a deliberately two-sided wedge: a useful 3D product now and the first public step toward general world models. It accepts text, one or multiple images, generates editable 3D worlds, and supports precise camera placement and export; emerging use cases include gaming, VFX, film, interior design, and potentially robotic simulation. World Labs intentionally tried not to make it a pure “science project,” while Fei-Fei Li calls Marble “the first glimpse” of a larger spatial-intelligence stack.
The current technical choice—Gaussian splats—turns generation into navigable geometry rather than a sequence of video frames. Splats render in real time on mobile and VR, although targeting 30–60 fps at high resolution on a four-year-old iPhone caps density and fidelity. Other systems already use frame-based generation, and future systems could attach mass or springs to particles or use token-based representations; the data structure is not treated as permanent.
Physics is the largest unresolved capability gap and the boundary between a creative tool and trusted engineering software. Pattern fitting might predict plausible orbits without deriving force vectors or “F equals MA”; Fei-Fei says there is “no indication” latent modeling yields causal law, while Justin hopes emergent physics appears at scale. Plausibility is enough for a film backdrop, but not for a building that must stand.
The commercial expansion path is horizontal, but its ordering remains intentionally unsettled. Creative industries are Marble’s beachhead; interior-design beta users already reconstruct and edit rooms, while robotics could use generated worlds as the “important middle ground” between scarce real-world data and uncontrollable internet video. Fei-Fei says whether to move directly into embodied use cases “is to be decided.”
World Labs’ discussion does not call for throwing out transformers; it points toward multimodality and a richer learning loop. Attention remains, and Justin notes that transformers natively model sets—the 1D order comes from positional embeddings—so spatial tokens need not require architectural demolition. The deeper missing ingredient may be hypothesis, action, falsification, and online updating, not a change of modality alone.
The talent and research bottleneck extends beyond capital: academia is under-resourced and too tempted to imitate frontier-lab scaling. Fei-Fei defends open benchmarks such as BEHAVIOR; Justin wants academia pursuing “wacky ideas,” including distributed primitives beyond matrix multiplication as clusters replace single GPUs. His hardware warning is concrete: Hopper-to-Blackwell performance per watt is “about the same,” leaving room for 10–20-year architectural bets.
Deep dive
Not yet available upstream; scheduled sync will retry.