Summary

Mirzadeh’s core claim is that benchmark achievement should not be mistaken for intelligence. A caveman might score near 0% on MMLU or GSM8K while current models reach roughly 80%-90%; Aristotle would also likely underperform current LLMs on those benchmarks. Yet Mirzadeh does not infer that the models are more intelligent: humans can learn, acquire skills, and create knowledge over time. The better measure is adaptability: “Intelligence is the slope of scaling, not the point at which you currently are.”
GSM-Symbolic finds instability where headline math accuracy suggests reliable reasoning. Template-generated variants produced gaps around 14%-20% for some models, with substantial variation. GSM-Names separately changes proper names without changing the numbers, while GSM-NoOp adds an irrelevant clause and causes a large performance drop. The host notes that eight-shot examples did not reliably make models filter out the distractor. Mirzadeh would prefer a model whose accuracy falls 10% consistently to one that loses only 1% but changes behavior arbitrarily.
Tool use can make an AI product more capable without showing that its underlying model understands anything. A chess engine can supply winning moves, but an intelligent system using it should eventually extract theories—why controlling the center matters, why e4, d4, or c4 help control more squares and develop pieces—and transfer those abstractions elsewhere. As tasks require five or ten interacting tools, orchestration itself becomes the original planning problem again.
Current objectives reward matching distributions, not acquiring concepts or beliefs. Cross-entropy only requires that “2 + 2 =” be followed by 4; it does not care whether the model understands natural numbers, addition, or why 4 follows. Prompting merely conditions or “nudges” learned distributions, so prompt dependence and distractor brittleness count against claims of reasoning.
The scaling thesis remains difficult to falsify because every failure can be answered with demands for another order of magnitude of capital. Even if a 10-trillion-parameter model trained on 100 trillion tokens fails outside its data, proponents can claim that 50 trillion parameters will unlock emergence. Mirzadeh does not oppose scaling, but argues that current methods have a novelty-learning slope “close to 0”; optimizing them may resemble making an airplane faster when the destination is the Moon.
A credible intelligence test should measure rapid learning of genuinely novel tasks, not mastery of familiar corpora. One proposal is to invent a programming language and compare how quickly a model and an unexposed human learn it; time matters because infinite search can eventually reach many goals. Agency is also important: learning requires “active engagement,” including choosing what to investigate next rather than passively absorbing supervised examples.
Mirzadeh has not ruled out Transformers or pure neural computation in principle; his criticism is about present representations and training. Internal neural activations might become symbols, and Turing completeness may be a necessary condition, but intelligence still needs an integrated world model that can form, challenge, and update beliefs. For investors, the practical warning is that saturated benchmarks are “a frozen cut of reality,” not proof that real-world autonomy or durable reasoning has been solved.

Deep dive

1. Benchmark scores confuse achievement with intelligence

Mirzadeh’s foundational distinction is asymmetric: an intelligent system should eventually achieve difficult tasks, but success on a task does not establish intelligence. Achievement measures present performance in a fixed domain; intelligence concerns how efficiently capability can grow.
His caveman test makes the mismatch concrete. A prehistoric human might score near zero on MMLU or GSM8K against roughly 80%-90% for current models, yet Mirzadeh rejects the conclusion that the models are more intelligent.
Aristotle sharpens the same point: he would probably underperform modern LLMs on contemporary benchmarks, but Mirzadeh still would not infer that the LLMs are more intelligent. “Intelligence is not about how much you achieve on a certain task.”
The host connects this to adaptation and skill-acquisition efficiency. Mirzadeh’s scaling-law version is memorable: intelligence is not a species’ current point on a brain-versus-body-mass chart, but “the slope”—how rapidly it improves when given experience and time.

2. Prompting and tools can raise performance without creating understanding

Asked whether brittle, distractor-sensitive behavior should disqualify LLM reasoning claims, Mirzadeh’s short answer is “yes.” He sees prompting as conditioning an enormous collection of learned distributions, steering the model toward one region without demonstrating that it understands the knowledge behind its answer.
Tool use is legitimate—humans use tools—but cannot be evaluated solely by whether the combined system completes a task. Real problems may demand five or ten complex tools whose outputs and state transitions must be coordinated; that orchestration “will reduce to the same problem” of planning.
Chess supplies his decisive specimen. A model relaying engine moves might win without knowing what constitutes a strong position, while human grandmasters use AlphaZero and other engines to develop novel openings, strategies, and explanations rather than merely memorize moves.
His proposed test is whether prolonged engine access yields theory. The system should infer why opening moves such as e4, d4, or c4 control the center, provide access to more squares, and prevent an opponent from developing pieces—not merely reproduce their empirical frequency.

3. The training objective rewards matching, not concepts

Mirzadeh’s diagnosis begins with distribution fitting: training minimizes distance from a data-derived distribution, then rewards staying within it. Asking what lies outside that “box” is therefore at odds with how the system was constructed, making distribution shifts predictably destructive.
Arithmetic exposes the missing requirement. Cross-entropy rewards producing 4 after “2 + 2 =”, regardless of whether that output comes from memorization, context, or an understanding of natural numbers and addition. “All the model has to do is give me 4 after ‘2 + 2’.”
He explicitly lacks a finished replacement, but argues that research often searches for solutions before forming a coherent model of the problem. After reading 10 or 20 incremental papers, readers may still be no closer to understanding how Transformers work.
A study showing that tuning a model raises accuracy by 5% exemplifies the problem: it is an empirical effect without an explanatory framework. He also worries that current theory may be too rigorous for the moment, while peer review remains noisy and difficult.

4. Novelty, speed, and agency are the better foundations for measurement

Mirzadeh offers no concrete intelligence metric and keeps that uncertainty intact. His starting point is axiomatic: define desired properties—especially performance on novel tasks—then progressively formalize what “novel” means rather than equating intelligence with whichever benchmark remains unsaturated.
A better coding test would invent a programming language from scratch and measure how quickly a model learns to write programs in it. Familiar-language accuracy mostly measures prior exposure; learning speed permits comparison with a human who has never encountered the language either.
Drawing on Jill G. Knack et al.’s psychology paper, “On the Definition of Intelligence,” he frames intelligence as a system’s maximum capability and capacity to achieve a novel goal within some amount of time. Time is load-bearing: with infinite time, many programs and systems can eventually reach a goal, but that says little about the system’s scaling slope.
Agency belongs inside that definition. Human learning depends on “active engagement,” described as necessary rather than merely beneficial: an intelligent agent should ask what to learn next, notice incomplete understanding, explore, and later exploit. On that account, supervised learning alone cannot suffice.

5. Abstraction allows knowledge to escape its original domain

The host challenges Mirzadeh with AlphaZero: if repeated play makes a particular strategic move emerge as a stable behavioral mode, why not treat that as reasoning? Mirzadeh concedes that the system learns state values and can act rationally within chess, but disputes that it forms a portable concept.
Human knowledge of “controlling the center” generalizes beyond four central squares. It can become control of strategically important positions—a road, passage, or other bottleneck—whereas AlphaZero’s world remains the particular game. That transfer is why abstraction lets humans “scale very quickly.”
Mirzadeh calls an abstract world model nearly indispensable, yet says the field cannot even compare basic representations. A function can be stored as an input-output table or as a polynomial; both return the answer, and the table may even offer the faster lookup.
Calling the polynomial “compressed” does not settle the issue, because encoding it presupposes concepts such as functions, powers, continuity, and real numbers. A friend suggested that one representation might be “more beautiful” than another, but the field lacks a formal measure of that beauty.

6. Symbols may emerge neurally, but beliefs need an integrated loop

Mirzadeh does not insist on a permanently separate symbolic module. Symbolic and connectionist elements might begin separately and become more integrated, but treating one as a detached external tool recreates the problem of action without understanding.
His missing component is a belief system that can evaluate incoming claims. If told an incorrect formula for a circle’s area, a system should compare it with its existing model, question the claim, and potentially reject it—not absorb every statement as equivalent conditioning data.
Beliefs should also have different standards for revision. A mathematical claim warrants doubt until supported by a proof; a recommendation for the best place to eat in Vancouver can be tried and readily updated. The desired system must know whether it needs to update its model.
On pure connectionism, his position is deliberately permissive: an internal activation could represent a symbol, and he sees no theoretical reason a neural system could not eventually create and compute over symbols. Turing completeness may be a necessary condition, although other architectures might be better than a Transformer.

7. More inference and scale may improve the wrong vessel

The host’s pushback cites chain-of-thought prompting, program induction, and active test-time fine-tuning as techniques that already compose behaviors and improve generalization in particular domains. Mirzadeh questions whether these gains show that a suitable underlying representation already exists.
His honest uncertainty is: “Is there anything there, beyond some interpolation between distributions?” If yes, these methods may build on it. If not, increasingly elaborate inference procedures may be ad hoc extensions of a foundation that lacks the object they are meant to improve.
The airplane analogy carries the conclusion: lighter materials, faster engines, better wings, and improved runways can optimize flight, but “it doesn’t necessarily help you achieve your goal” of landing on the Moon. Before extending the vessel, researchers should ask whether it can reach the intended destination.
Scaling remains empirically slippery. A failed model with 10 trillion parameters and 100 trillion tokens can always be answered by predicting emergence at 50 trillion parameters, beyond an opponent’s experimental budget. Mirzadeh’s alternative is to compare learning slopes; he currently judges them “close to 0” across the architectures and methods available.

8. GSM-Symbolic exposes instability beneath high math scores

GSM-Symbolic was developed while the researchers were evaluating reasoning systems, exploring ways to improve reasoning, and considering more computation per output token. Template-generated variants were intended to support robust evaluation and perhaps better training data; a sanity check against GSM8K instead revealed gaps and striking variance.
Some models showed reported gaps around 14%-20%. The most surprising result was not the absolute accuracy but the variance across models. Mirzadeh proposed a human cross-evaluation that would change words such as “Amy” and “apples” to “John” and “bananas” while preserving the logic. GSM-Names separately changes only proper names, without changing the numbers or adding anything to the question.
The host notes that frontier models still retained nontrivial performance, including o1 on a related renamed-symbol planning test. Mirzadeh responds that undisclosed synthetic or human-created math data makes prior exposure impossible to exclude.
GSM-NoOp adds a clause carrying no relevant logical or arithmetic operation, so ignoring it should preserve the answer. Performance dropped heavily. The host notes that eight-shot demonstrations still did not make models reliably filter out the distractor; intermediate variants with one or two genuinely operational clauses were used to examine performance between these conditions.

9. Frozen benchmarks and majority votes conceal the failure mode

Mirzadeh would rather see accuracy decline 10% on harder questions with stable reasoning than decline 1% while varying arbitrarily under irrelevant substitutions. The central question is not “How many answers were right?” but why a supposedly understood problem changes when the wording changes.
Benchmark saturation encourages the field to replace GSM8K near 95%-100% with a harder target such as FrontierMath, then drive performance from roughly 10% toward 90%. That cycle measures optimization against another fixed sample, not the creation of knowledge or robust understanding.
His computer-vision analogy is cautionary: ImageNet benchmarks were saturated and vision appeared solved, yet self-driving remains difficult because reality is not “a specific frozen cut” with a fixed set of examples. Real environments change, demanding an agent that understands and reasons.
Sampling does not rescue the epistemology. For arithmetic, “2 + 2 =” should deterministically produce 4, so the study used greedy decoding; a paper reported that sampling a model 100 times on GSM8K and MATH may improve performance by 20%, but this resembles sending “10,000 drunk people” home and celebrating that some eventually arrive. Successful search is not understanding.

28. Inside GTC 2025 (Part I) | Nvidia's History: How One Company Kept Betting on the Future for 30 Years a16z on AI Voices: Call Centers, Coaches, and Companions with Olivia Moore & Anish Acharya