No Priors Ep. 135 | With Humans& Founder Eric Zelikman

Summary

• Zelikman’s core research claim is that reasoning can be trained as an iterative competence, not merely elicited by prompting. STaR generates solutions, learns from those reaching the correct answer, and repeats; on n-digit arithmetic, more iterations kept extending the solvable length, with “no obvious plateau here.” Training on failed problems through rationalization—or, as later work suggests, sampling far more—helps escape the positive-example ceiling.
• Quiet-STaR pushed that recipe from question-answer tasks toward pre-training-scale text, opening a path to more general reasoning. Its overlooked ingredients included online learning and a difficulty-sensitive baseline: for harder problems, learn more, and for easier problems, learn less. This was similar in some ways to later RL-pretraining work. After pre-training data for Grok 2, the reasoning recipe for Grok 3, and tool-use and agent infrastructure for Grok 4, Zelikman still sees “meaningful dimensions” of IQ scaling left.
• Current frontier models can solve pockets of work that are nontrivial for PhD researchers, but remain jagged and distribution-bound. Humanity’s Last Exam includes genuinely difficult questions, many of them traps that punish assumptions; performance depends heavily on supplied context, proximity to the RL distribution, and whether answers are closed-form and verifiable. Providers’ undisclosed task-specific RL data makes precise diagnosis difficult.
• Agent performance partly reflects a product trade between deliberation time and responsiveness, not just raw intelligence. Background systems such as Codex can deliberate longer than interactive tools such as Cursor, while verifiability generally helps models more than non-verifiable work. For investors, inference budgets, interaction mode, and reward design therefore belong inside the capability thesis.
• The industry’s favored autonomy metric may extend task horizons while eroding user agency. Zelikman cites METR-style movement from roughly two-hour to 2.5-hour autonomous tasks and imagines an eight-hour agent returning an artifact that is only “somewhat there”; he says people may then have less real agency over, and understanding of, what they are building. Sarah stresses that keeping people in the loop is an “active decision,” not an automatic by-product of scaling. Elad’s joke—“20,000 lines of generated code looks good to me”—lands alongside Eric’s example of 100,000-line generated pull requests.
• Humans& is betting that the neglected axis is not companion-style EQ but models that understand people well enough to improve productive collaboration over time. Today’s task-centric benchmarks support organizational credit assignment—one team does 2% better, or improves a benchmark by 10% versus another’s 5%—and easy-to-grade RL, but they train systems to stuff everything into one turn rather than ask, clarify, express uncertainty, or act proactively. Long-horizon effects, including sycophancy and recently discussed psychosis-related issues, remain poorly handled by this regime.
• The economic thesis is expansion, not merely automation: models that understand goals could “grow that pie” instead of carving up existing GDP categories. Zelikman believes hard problems are more likely to yield to models coordinating large groups across differing goals, values, ambitions, and weaknesses than to an AI disappearing for 20 hours and returning with “the answers to life, the universe, and everything”—though he explicitly says “we’ll have to see.” Memory is underinvested: without it, current models resemble a friend who remembers only your name and a high-level sketch of your life.

Deep dive

1. Reasoning scaled when models learned from their own work

• Zelikman’s original motivation was unused human talent: automation could remove unwanted work, but empowerment ultimately requires understanding what people actually want. Yet around 2021 he chose capability first: language models were “not very smart,” and chain-of-thought prompting offered only a small improvement at that time.
• STaR’s mechanism was intentionally bare: generate solutions, reward and learn from those reaching the correct answer, ignore the others, and repeat as the model tackles harder problems. The original had no baseline and was compared with REINFORCE.
• The scaling clue came from what he recalls as n-digit addition or multiplication. With each training iteration, the solvable digit length rose: “there’s no obvious plateau here.” The absence of an obvious plateau was a major surprise and changed his intuition about scalability.
• To escape a positive-only local minimum, the team showed failed problems back to the model, asked it to reason about why it should have gotten them right, and trained as though it had succeeded; later work suggested that sampling many solutions could also surface positives. Quiet-STaR then aimed to scale reasoning to arbitrary pre-training text and next-token prediction, using online training and a difficulty-sensitive baseline—features Zelikman now regards as overlooked “nuggets.”

2. Frontier intelligence is jagged, contextual, and distribution-bound

• Sarah describes Zelikman’s xAI arc as pre-training data for Grok 2, the reasoning recipe for Grok 3, and tool-use and agent infrastructure for Grok 4. Asked for a human comparison, he says direct comparison is hard because intelligence is “very jagged.” Some Humanity’s Last Exam questions solved by models are nontrivial even for PhD researchers, though not open problems; many are assumption traps.
• His practical rule is to supply as much context as possible. Models are strongest when a question admits a closed-form numerical answer or an easily checked choice—“if you can imagine it being easy to check your answer,” it is generally easier for the model too.
• Responding to Sarah’s coding challenge about why sophisticated tasks still fail, Zelikman splits the answer between latency and distribution. Background Codex can deliberate longer than interactive Cursor, while performance falls as tasks diverge from training. Without visibility into each provider’s RL data, precise diagnosis is impossible, and non-verifiable tasks still lag verifiable ones.

3. Autonomy is becoming both the metric and the product risk

• Zelikman still sees meaningful IQ-scaling dimensions left. Sarah emphasizes that keeping humans in the loop is an “active decision,” and he agrees. Asked whether the case is a higher capability ceiling, more efficient error correction, or users’ philosophical preference for involvement, he says it is probably some combination.
• Sarah suggests that labs may prefer more rollouts to recruiting people for complex, out-of-distribution reasoning traces because people are operationally messy. Zelikman notes that, regardless of the incentive, autonomous horizon is increasingly used as a progress metric.
• He points to recent or recentish IMO systems reasoning for hours and the METR-style headline moving from roughly two-hour to 2.5-hour tasks without intervention. He hedges that the numbers may not deserve face-value treatment, but notes that labs increasingly look to them.
• The product cost appears when an eight-hour agent returns something only “somewhat there”: Zelikman thinks people may have less real agency over, and understanding of, what was built. Elad’s “20,000 lines of generated code looks good to me” meets Eric’s 100,000-line generated pull request, capturing why he expects generated systems to become increasingly opaque to their users.
• His economic alternative is not merely carving world GDP into automatable, x-billion-dollar markets. If models understand goals and empower people into out-of-distribution activity, they may “grow that pie” and produce more real innovation than systems designed to replace each existing chunk of work.

4. Humans& changes the objective from finishing tasks to helping people

• Sarah frames EQ as emotional or interactive capability that has often appeared in character and companionship products; Zelikman reframes it as enablement. His earlier PhD work trained language models to simulate different students and showed that those simulations could help design better tests—evidence that modeling people can improve systems for them.
• Zelikman recounts being told by people at Google that easily quantified benchmarks help with credit assignment: one team can do 2% better than another, or improve a benchmark by 10% while another improves it by 5%, making resource allocation legible. This also fits the easiest ways to train models. Elad notes that the most popular environments are coding and computer use; very few benchmarks ask how sustained interaction affects a person’s life.
• Treating each turn as a finished game teaches the model to cram every possible answer into one response, not ask, clarify, express uncertainty, or act proactively. Zelikman ties this task-centric regime to issues such as sycophancy and recently discussed psychosis-related harms; praise the “gloves for catching ice cream” startup today, and the user may build it, discover it was a bad idea, and stop trusting the model.
• Academic work such as RL from human interaction and CollabLLM is beginning to use interaction or simulation, but he says industry pays far less attention. With the caveat that this may be a strong statement, he says that for most labs the human is “the intermediate” before a fully automated system, making deep collaboration look temporary.

5. Memory turns a tool into a collaborator

• Zelikman says the five-year vision need not take that long: in the nearer term, a model helping find a Paris hotel for a wedding could already know the event, companions, constraints, and other relevant context. Memory is “underinvested,” but isolated-task training rarely rewards cross-conversation information; the hosts compare current models to a friend who remembers only your name and a high-level sketch, forcing users to restate relevant context.
• Elad’s “unique snowflake” objection preserves the hard part: people want to learn without doing the work, eat cake while getting fit, change their minds across time scales, and remain “constant distribution shift.” Zelikman concedes personal models will not be perfect—“I’m not going to pretend that I’m going to one-shot this problem”—but says an objective that actively learns the person would already go far.
• Against Culture-style abundance, Zelikman wants models that “empower us”; he thinks problems like curing cancer are more likely to yield through coordination across goals and values than through a 20-hour oracle returning “the answers to life, the universe, and everything,” though “we’ll have to see.” Humans& is hiring builders in memory research, distributed systems and fast inference, plus creative, “beautiful, tasteful” product design; reflecting his previous company’s approach, he wants everyone to be, to some extent, an engineer.