165: GEAR 高深远: World Models, Self-Evolving Loops, DreamDojo
Summary
高深远的核心判断是，机器人自进化不必等到“完美世界模型”：policy、action-conditioned world model，以及负责规划和打分的通用 agent，只要把级联误差压到可接受阈值，闭环就能从局部任务启动。 录制时他判断，苗头“可能就发生在今年”，即 2026 年；但他同时估计 WAM 还需 2—3 年成熟，DreamDojo 式通用模拟器真正成为大爆点则要约 5 年。因此，必须区分“简单任务首次跑通”和“大规模真正有用”。“一旦连接起来，整个循环会变得越来越简单，越来越自动。”
在粒子、显式 3D、隐空间和纯视频四条世界模型路线中，高深远最看好 video-in/video-out，因为语言和视频是目前最具代表性的两个 data-rich 模态，也最容易承接基座模型的泛化能力。 显式 3D 的坐标一致性强，却要经历重建、渲染和 3D 标注；JEPA 式隐空间更紧致高效，却暂时难以直接接入 GPT、Gemini、Sora 等生态。视频路线的关键资产不是单一算法，而是强视频基模、训练分布、算力 infra 与机器人 action 数据的组合。
DreamZero 与 DreamDojo 不是互相替代的两种产品，而是闭环中的 policy 与 simulator：前者接收 WAM 的任务文本，同时输出未来视频和机器人 action；后者接收真实 action 轨迹并预测未来世界状态。 DreamZero 的功能定位与 VLA 对等，但多了视频预测；DreamDojo 则可以在动作真正执行前比较后果。高层任务可以先在文本步骤上搜索，低层抓取角度、力度和速度再由 DreamDojo 优化，形成“想步骤—看未来—调动作”的两级推演。
世界模型近期最可兑现的价值，是把具身智能最昂贵、最不公平的物理环节转化为可并行计算：部署前搜索、可复现评测、虚拟遥操采数、干预纠错和 reinforcement learning。 现实测试受人员、光照、摆位、校准和物品损耗制约，模拟环境则可以精确重置；如果速度和真实性继续提升，训练轮数取决于 GPU 数量，而不再取决于“一天二十四个小时”。高深远将其概括为“拿算力换高效性和公平性”。
真正卡住闭环的不是生成画面够不够漂亮，而是物理理解与 action control 两种泛化能否同时可靠。 只有专家轨迹会让世界模型形成错误偏好：即使抓偏了也生成抓成功，模型便失去反馈区分度；它需要做到“你给它一个抖的，它就是得抖，你给它一个丝滑的，它就是得丝滑”。好消息是，它不必覆盖整个动作空间，只需覆盖当前 policy 可能输出的分布；而 policy 越强，这个分布越窄。
DreamDojo 的实质性进展，是以 Cosmos 为基座，用约 44K—45K 小时 egocentric 人类视频预训练，最后再用机器人数据微调，并通过 Latent Action 把无动作标签的视频转化为可控训练信号。 团队称其对机器人数据未见过的环境和物体有更好的泛化；但测试物体至少在数据中出现过，或存在类似物体，数据分布也未经精细调配。所谓 OOD 是相对于机器人数据而言，而不是相对于整个预训练语料。Latent Action 是否长期必要，高深远本人也“打上一个问号”。
这个市场暂时没有统一 benchmark，投资人很难从论文表格判断领先者：不同机器人的自由度、action space、相机位置乃至改装方式，都会让模型无法直接 zero-shot 比较。 当前团队往往要把别人的世界模型重新训练到自己的本体上，因此只能比较少数模型；最终可能靠一个跨本体世界模型，或行业收敛到少数标准机器人来解决。DreamDojo 因内部已有傅利叶 GR-1 数据而选择该本体，并非对其商业或技术优劣作出背书。
竞争胜负仍会高度依赖视频 foundation model，但不会简单等同于“谁的视频最多”：电影级生成所优化的分镜与分布未必服务机器人决策，面向 Physical AI 的预训练倾向更关键。 高深远最关注 Google DeepMind 的 Genie 3、SIMA、Veo 闭环，也认为 NVIDIA Cosmos 的开源路线可能降低 startup 的前置成本。他提到 OpenAI 将 Sora 团队重组到 robotics，认为这可能意味着其重新认真推进相关工作。创业公司仍可从 reward agent、垂直 policy、世界模型或特定 domain 切入，但如果无法掌握基模，便会在训练分布、基础设施支持与版本时滞上处于劣势。
Deep dive
1. A Decision-Relevant World Model Must Control the Future Through Action
高深远 first strips away the buzzword: a multimodal foundation model with extensive world knowledge can call itself a world model, but the stricter definition in decision-making is a model that predicts the future from historical states, with those predictions conditioned on a given action.
Sora’s description of “video generation as world simulator” holds under this broad definition: text can edit the sky, a person, or the surrounding environment, and is therefore also “an intervention on the world.” But controlling pixels with text is not the same as accepting a robot joint trajectory, a vehicle path, or a game character’s action.
A model that is genuinely useful for decision-making must answer not “what does the user want to see?” but what the world will look like after an agent executes a specific action. “Action is an intervention on the world.”
2. A General Agent Must Initially Assume It Controls Only Itself
程曼祺 pressed on the difference between the agent behind text-based video editing and a robot: a video user changes the environment like “God,” while a robot operates from inside the world. 高深远 acknowledged that this is a critical distinction; a general-purpose embodied policy should typically control only its own actions.
Editing other agents or the environment still has value, for example in constructing rare cases for autonomous driving or games. But a robot deployed into an arbitrary environment cannot assume it can communicate directly with other agents, so “from the perspective of generality, a world model of the self is still the most general.”
Multi-agent systems could ultimately outperform humans: vehicles and robots can communicate directly over networks, and V2X can improve both safety and decision quality. The consensus was to first make single-agent autonomy work, then consider networked world models for V2X or 10B robots communicating with one another.
3. Particles and Explicit 3D Trade Structure for Efficiency and Consistency
The most abstract approaches represent object states with particles, points, graphs, or geometric matrices. They are cheap to simulate and require less data, but flexible objects and other categories often need bespoke definitions, limiting cross-object generalization.
World Labs’ Marble represents the explicit-3D route: it reconstructs the world through point clouds, Gaussian Splatting, occupancy representations, or similar methods. Object coordinates are explicit, making movement, editing, and spatial navigation easier, while long-term consistency is naturally stronger.
The cost is a multistage “observation → reconstruction → rendering” pipeline. It typically requires 3D labels and is not fully end-to-end. The policy still consumes images or video, so every additional representation conversion adds another data and optimization bottleneck.
4. JEPA Compresses Decision-Relevant Information; Video Preserves the Existing Foundation-Model Ecosystem
The JEPA-style approach promoted by Yann LeCun/AMI predicts the future in latent space, retaining only the high-level semantics most directly relevant to decisions. The classic example is that a person walking down the street need not predict every face; knowing there is “roughly a person” ahead is enough to avoid it.
高深远 agrees that latent space is more compact and cheaper to run, especially for deployment on real robots. But the system first has to learn the right space, then requires its own evaluation framework and fresh alignment with policies, language models, and video models.
程曼祺 summarized the limitation as difficulty benefiting directly from industry progress: a new latent space cannot naturally be read by GPT or Gemini, nor can it plug directly into a Sora-style model. 高深远’s response was that the route “has promise,” but language and video may already be sufficient paths toward AGI.
His own bet is video-in/video-out. Pixel prediction is only the training target; decision-making does not require commitment to one exact detail. Generated outputs are samples by nature, and modern models may even be better than humans at filling in missing visual content, so reconstruction objectives and decision usefulness are not inherently in conflict.
5. The Video Route’s Core Advantage Is Aligning Rich Data With Scarce Action Data
高深远 believes the scalable path toward AGI should start from the two data-rich domains of language and video, then align them with the scarce modality of robot action. Building an entirely new representation would forfeit the generalization and engineering base accumulated by existing foundation models.
GEAR Lab is not betting exclusively on video and will explore any promising route that matches its research interests. But regardless of the eventual prediction space, the latent space itself is usually learned from video, while visual reasoning speed should continue to improve with better chips and generation models.
6. World Models Took Off Once Generation, Data, and Policies Cleared the Starting Line
The first catalyst was Sora-style generative models. The industry moved from “nothing can be simulated” to generating realistic scenes and some physical regularities, turning world simulators from a concept into an object of experimentation.
The second was data supply. Policies mainly need expert trajectories, while a world model must simulate successful, failed, and arbitrary actions without bias. As embodied-data vendors, open datasets, and action data from different distributions have expanded, training a simulator has become feasible.
The third was the policy itself. Early policies could handle only simple lab tasks, had little need for generalized feedback, and produced excessively chaotic actions. Today’s action distributions are more controlled, making them easier to simulate; the policy’s next priority has shifted toward generalizing across scenes and tasks.
7. Policies and World Models Feed Each Other Action and State at Every Timestep
A policy can be any decision model, including a VLA: it reads an observation and outputs an action. The world model reads the historical state and that action, predicts the next observation, and returns it to the policy for another round of interaction.
Both components can run on the robot or in the cloud when network conditions allow. There is no absolute policy/world-model boundary based on model size; the key constraints are the deployment environment, latency, and bandwidth.
高深远 stresses that a world model does not need to fairly simulate every action in theory. It only needs to cover the distribution the policy might produce. As the policy matures, actions become more like “when you pick up a cup, you pick up a cup,” and the simulator’s effective search space narrows.
8. DreamZero Is a Policy With Video Prediction; DreamDojo Is the Action-Conditioned Simulator
DreamZero is called a World Action Model: it takes the current task’s text instruction and predicts both future video and robot actions. Functionally, it remains a policy equivalent to a VLA and is generally used as one.
It also has a world-model component. If text planning is treated as a high-level action, DreamZero generates the corresponding future state under that text condition. Compared with a VLA, it adds video prediction; compared with an ordinary world model, it directly outputs low-level actions.
DreamDojo instead takes real robot trajectories and other action inputs and generates the corresponding future video. It is the stricter action-conditioned world simulator.
WAM therefore does not simply “replace the world model.” DreamZero’s proposed actions can be passed into DreamDojo, which simulates their consequences before execution. The two systems handle proposal and verification, respectively.
9. High-Level Text Search and Low-Level Trajectory Search Form a Two-Level Rollout
When people perform a task, they usually think first in steps such as “pick up, move, put down,” rather than planning every joint directly. DreamZero can generate future states for multiple textual subtasks, allowing a person or agent to select the step most likely to complete the objective.
Once a subtask is selected, DreamZero has already produced a robot trajectory, but approach speed, grasp angle, and force can still be optimized. Those low-level action candidates can then be passed to DreamDojo to compare their respective future consequences.
高深远 explains the value through human imagination: “Without a world model, every decision ignores the consequences—you do it first and deal with the result later.” A world model moves real-world trial and error to before the action is executed.
10. World Models First Improve Safety and Make Embodied Evaluation Reproducible
Imagination-based search lets a robot see the potential consequences of an action before choosing it. Its most direct value is therefore greater safety and generalization, not merely more realistic video generation.
Real-world evaluation is both inefficient and unfair. It requires physical robots and staff on site, and failures consume objects. Lighting, object placement, and sensor calibration cannot be identical in the morning and afternoon, while different labs cannot reproduce the same room.
A world model can save any initial state and reset it precisely, allowing the same set of policies to interact repeatedly under identical conditions. 高深远 calls the essence “using compute to buy efficiency and fairness.”
11. Real-Time Simulation Moves Teleoperation, Correction, and Data Collection Into the Virtual World
Once DreamDojo runs in real time, an operator can use existing teleoperation hardware to control a simulated robot inside the world model and record the corresponding actions without owning a physical robot.
Once a policy is basically functional, the operator can intervene during a simulated run and correct an error in progress. Those intervention trajectories can be used to continue training the policy, producing more targeted data than a demonstration from scratch.
Teleoperation hardware is still required, although wearables can simplify it. The main savings come from eliminating the robot body, physical site, reset time, and wear and tear—not from removing the human operator altogether.
12. Virtual Reinforcement Learning’s Biggest Lever Is Breaking Through Physical Time
A real robot has only 24 hours a day and is constrained by staff schedules and the number of available machines. A world-model environment can run at GPU speed and scale horizontally with additional cards, allowing a policy to trial and error repeatedly without damaging real equipment.
高深远 uses QQ Farm as an analogy: people sleep while the virtual world keeps moving. If the reality gap becomes small enough, running for a month on a cluster “might produce something emergent,” but he explicitly says the field is “still far from that stage.”
程曼祺 cited Demis Hassabis’s vision of using Genie- and SIMA-style systems to accelerate scientific experiments and even controlled nuclear-fusion research, then raised the chicken-and-egg question: if the simulated world is already realistic enough, would AGI or embodied intelligence not already have been achieved?
高深远’s answer is that the loop does not need to wait for a perfect simulator. Locally reliable feedback can already improve a policy, and the improved policy can generate a better training distribution. The world model and agent can therefore improve in a rising loop rather than arriving one after the other.
13. The Self-Evolving Loop Combines a Planning-and-Evaluation Agent, a Policy, and a World Model
A general agent defines the task, passes text plans to the policy, and judges whether the future state generated by the world model is good. The judgment can take the form of a reward score or textual feedback.
The policy outputs an action, the world model uses it to generate future video, and the agent evaluates the result and updates the policy. The three interfaces are text, action, and video—precisely the modalities that are currently more mature or data-rich.
The loop currently works only on simple tasks because the agent’s reward generalization, the policy’s action-proposal generalization, and the world model’s future-prediction generalization can all fail, with their errors compounding into a cascade.
14. Once the Error Threshold Is Crossed, the Loop Accelerates as the Action Distribution Narrows
Once a policy reaches a basic level of safety, it can enter a new room and collect data on its own rather than immediately flailing, breaking objects, or damaging the arm. Even mediocre data will fall within an acceptable action range.
That automatically generated data helps the world model learn the two most important capabilities: “simulating physics” and “controlling through action.” The more realistic the prediction and the more precisely it corresponds to the input action, the more reliable the feedback to the policy.
A stronger policy in turn produces a narrower, more rational action distribution. The world model no longer has to handle as many jittery or absurd actions; as simulation becomes easier, feedback becomes more accurate, creating a positive flywheel.
高深远 believes the critical point for simple tasks “might happen this year,” meaning 2026. But he separately estimates that the WAM architecture and training approach need 2—3 years to mature, while an action-conditioned world model in the DreamDojo mold could become a major breakout in about 5 years. These are judgments at different levels, not one launch schedule.
15. The Ceiling on Generalization Is Set by New Physics and New Actions
高深远 ranks the current research priorities as generalization, long-horizon stability, and efficiency. Generalization “determines the upper bound of where the world model can be applied,” and he further divides it into physical understanding and action control.
Physical generalization means understanding objects such as water, paper, and cloth in scenes not covered by robot data. If a soft cloth is generated like a rigid body, the video can be as sharp as desired and still provide no credible feedback.
DreamDojo’s main objective is to transfer scenes, object interactions, and physical knowledge from human video to robots. Only once the simulator covers new environments can a policy improve in those environments through feedback.
16. Expert-Only Trajectories Teach a World Model to Generate Failure as Success
Policies, including VLAs, need optimal demonstrations, so existing robot datasets consist largely of expert trajectories where “the grasp succeeds.” A world model, by contrast, should simulate all relevant actions without bias and should not favor successful outcomes.
That data bias creates a fatal false feedback signal: given a trajectory with a poor grasp, the model still generates a successful grasp because it has never seen failure. “If you give it a shaky action, it has to be shaky; if you give it a smooth action, it has to be smooth.”
First-person human video contains many casual and non-expert actions, expanding the action distribution. Current policies can also automatically collect the actions they are likely to produce. As policies improve, the world model will ultimately need to cover an increasingly narrow space closer to expert behavior.
17. Long-Horizon World Models Need Stable Generation and Memory for Occluded Objects
Multi-round interaction in generative world models accumulates error, while explicit 3D is more stable because absolute coordinates remain fixed. Efficiency remains an issue, but 高深远 believes both problems align with the demands of cinematic video generation and will be pushed forward by the same mainstream technologies.
Current attention may be on short-horizon, roughly 1.5-second instantaneous decisions, while household service involves continuous long-horizon tasks. The model must keep the scene from drifting and preserve the state of objects while they are temporarily out of view.
A typical failure is putting an item in a cabinet, closing the door, and opening it again. Existing generative models may randomly decide whether the object is still there, giving the policy a random signal rather than reliable world feedback.
For cross-day household memory such as “which drawer contains the scissors,” 高深远 believes it is more reasonable to use text or long context to record the abstract location. Embodied research is currently more focused on producing high-success-rate actions for known tasks.
18. The Lack of a Unified World-Model Benchmark Comes From the Nature of Action Spaces
Nearly every paper currently builds its own benchmark and compares against only a handful of models. World models cannot be tested off the shelf like LLMs because their action inputs are tied to a specific robot’s degrees of freedom and dynamics.
Even when researchers use the same robot, such as a Unitree platform, the observation distribution changes with camera placement—for example, whether the robot head is removed and a camera is mounted on the neck. The same action space does not mean an existing model will work zero-shot.
A fair benchmark could emerge in one of two ways: a single world model could master multiple robot action spaces, or the research community could converge on one standard embodiment. 高深远 believes that if the field reaches the first state, the world model itself will already be very powerful.
For now, teams must retrain other people’s models on their own robots, at a cost far above calling a language-model API. Interim metrics include visual realism and action following, but the final test remains whether the model improves downstream policy success rates.
19. DeepMind and NVIDIA Are Assembling Closed Loops Through Rich-Modal Interfaces
The team 高深远 is most willing to follow is Google DeepMind: Genie 3 provides an interactive world, SIMA is the policy, Veo supplies the video foundation model, and a Gemini-style VLM can handle task planning and state scoring.
DeepMind has an initial real-world advantage from validating the stack in games. Game data can be generated, parallelized, and evaluated without physical-time constraints; game development also embeds substantial 3D and decision knowledge that may eventually transfer to robotics.
NVIDIA’s Cosmos is closer to the video and vision-language foundation-model layer and needs post-training to connect to action. On the embodied side, DreamDojo corresponds to the simulator, while DreamZero, a VLA, or another policy occupies SIMA’s decision role; a general VLM handles the connections and evaluation.
20. World-Model Competition Is Splitting Across Explicit 3D, Latent Space, Games, and Robotics
The main routes 高深远 identifies include NVIDIA’s video world models, Yann LeCun/AMI’s latent representation, World Labs’ explicit 3D for applications such as games, and Wayve’s Gaia series focused on autonomous driving.
He is also watching General Intuition’s game-world-model thesis: the large volume of game-behavior data accumulated during the pandemic could break through physical constraints. GEAR’s earlier EgoScale work likewise showed that decision knowledge learned in games can help real-world policies.
On OpenAI, he noted that the Sora team was moved into the robotics lab and inferred that the company may be trying to do something in world models; he expects competition to be intense in 2026. He gave no firm figure for the team’s size.
Anthropic has made few public moves in this direction, but 高深远 does not view that as necessarily a miss. Controlling virtual agents is already a powerful capability; physical AGI could first strengthen the multimodal foundation model and leave robot alignment for the final step.
21. Physical AGI Has Two Paths: Touch Robot Data Now or Align at the End
The first route trains robot policies, world models, and action data immediately. This is the path taken by NVIDIA GEAR and many embodied startups, with the advantage of directly addressing dynamics and real-robot problems.
The second route first builds exceptionally strong language, vision, and virtual-agent systems so the model learns general representations and world knowledge, then aligns vision with action. 高深远 believes ByteDance, AMI, and potentially Anthropic in the future could follow this approach.
Both routes make sense. DreamZero is attracting attention because it initially demonstrates strong alignment between the video modality and action, suggesting that teams with powerful foundation models will not be unable to enter robotics later.
22. 高深远 Shifted From V2X Data Generation to Single-Agent Autonomous-Driving World Models
He began studying world models around the second half of 2023, initially working on multi-agent perception and V2X. Most available data then came from single agents, so he used Diffusion image generation to supplement the scarce multi-vehicle data.
After Tesla released its world model, he shifted toward the more general single-agent route. GenAD attempted to collect autonomous-driving video from YouTube, breaking beyond the field’s reliance at the time on roughly 5 hours of public data from a single city, Singapore.
VISTA then continued the work on video-level physical quality, action-control precision, and control modalities. But he also saw academia’s disadvantage: automakers already had high-quality video and trajectory labels, while researchers still had to download and clean data and add action labels. “Academia as a whole is actually behind Tesla.”
23. Joining GEAR Was About Research Fit and Scaling Original Exploration
高深远 first connected with GEAR in 2024, but still wanted to spend more of his PhD years publishing as first author. He reconnected in 2025, joined as an intern, and will formally enter NVIDIA GEAR Lab after graduation.
One point of alignment was Latent Action. He had already been studying whether a latent-action policy could be connected to a latent-action world model, while the GEAR team was exploring a similar direction.
GEAR has one main line around policies such as GR00T N1 and N2, and another extending world-model work from DreamGen. 高深远 chose a DreamDojo-style action-conditioned simulator because he wanted to understand what world models could ultimately “enable.”
24. DreamDojo Trades Roughly 44K—45K Hours of Human Video for Robot Generalization
DreamDojo aims to provide reusable pretrained weights. When the community works with a new robot, it should not need to train from scratch; it should only need to connect its own action data and fine-tune to obtain the corresponding world model.
The training pipeline is a Cosmos video foundation model, pretraining on egocentric human video, and fine-tuning on robot data. The human-video stage used roughly 44K—45K hours. The team observed better generalization to new scenes and objects not covered by the robot data.
This scale was not a theoretically optimal point; it was all the data available for training the final version. 高深远 acknowledges that the team has not yet fully tuned the data distribution, and that further balancing and filtering “might make” transfer performance better.
Another advance is real-time performance. Earlier models often forced a choice between real time and generalization; DreamDojo is designed to satisfy both, allowing users to experience direct teleoperation and quickly search the consequences of actions during policy deployment.
25. Latent Action Unlocks Unlabeled Video, but 高深远 Is Not Sure It Is the End State
Latent Action extracts action representations from unlabeled video in a unified way without knowing the specific embodiment or viewpoint. In theory, “as much video as you have, that much video can receive action labels,” preserving the scalability of video data.
DreamDojo extends the approach he validated in AdaWorld. With limited resources at the time, he had more than 1,000 games continuously generate behavioral video, then used Latent Action to construct a controllable world model.
The representation is a noisy “way station.” It cannot precisely describe every body, but sits close enough to different action spaces to support cross-embodiment transfer. Like a person who loses both hands and switches to using their feet, the system still has to adapt from a shared intent to a new actuator.
But he deliberately retains some uncertainty. If both human and robot data eventually come with high-quality labels and embodiments become increasingly human-like, routing through Latent Action could prove unnecessary. Data collection is also shifting from dedicated setups to wearables that automatically capture labels during work, while synthetic data comes with labels by construction.
26. DreamDojo’s Evidence, Boundaries, and Next Steps All Point to Closed-Loop Generalization
The team chose 傅利叶 GR-1 primarily because it already had internal data and made test-set construction convenient, not because it was asserting that the platform was superior to other embodiments. Long-term selection will also depend on failure rates, ease of use, adoption, and proximity to human data and dexterous hands.
The 6 benchmark groups revolve around two axes: the realism and physical plausibility of future frames, and the precision with which different actions control the generated result. The team first validated on internal GR-1 data, then used multi-object datasets such as EgoDex to confirm transfer, before expanding the human-data component.
“Out of distribution” strictly means unseen in the robot training data, not unseen across Cosmos and the roughly 44K—45K hours of human video. 高深远 even believes scaling will eventually make “every problem a problem the model has seen”; the DreamDojo-HV dataset is expected to be released after subsequent research is completed.
In response to the view that “policy alone is enough,” he says world models are indeed harder but do not need to become perfect first. Even locally noisy feedback may improve a policy. A startup can test scaling laws through a reward agent, a vertical domain, or any individual component, but owning a video foundation model still provides advantages in training distribution, infrastructure, and iteration latency. His ultimate goal in industry is to keep raising the generalization ceiling of every component and truly connect the “self-evolving loop.”