Ep. 042 - GDPx: The Benchmark for Frontier AI | Andrew Megalaa, Jordan Nanos, Max Kan
Summary
- GDPX targets the domains its creators expect to capture the first $1 trillion of AI revenue. Max Kan’s thesis is that this revenue will be “highly concentrated within a few domains” that keep earning high returns from smarter models, including chip design, biology and cybersecurity. Most routine tasks, he expects, will not require frontier intelligence heading into 2027; GDPX therefore measures a deliberately selective slice of economic activity.
- The benchmark’s central distinction is between genuine difficulty and defective evaluation. Max calls it “difficult versus unfair”: a correct solution can fail because the verifier recognizes only another valid approach. SemiAnalysis sourced tasks from 14 specialized providers, rejected more providers than it admitted, and retained most tasks privately to reduce benchmark optimization.
- Materials science produced the clearest surprise about capability relative to cost. Astra beat a published state-of-the-art reference on one subtask, while contributor Brian discovered that some supposedly difficult workflows failed because he was “prompting it wrong” or providing insufficient context. Jordan Nanos highlights materials-science task costs reaching $40–$50, with some models solving tasks for about $10.
- Realistic finance and legal work remains substantially beyond reliable autonomous execution. The finance environment reconstructs a transaction involving more than $800 million for 62% of a snack company, with roughly 4,000 files. Finance was essentially zero under binary grading apart from an isolated Carbon success; Opus’s partial score was only 50-something percent, while the best legal partial scores were in the 30s or 40s.
- Safety restrictions materially change which model is useful, even when underlying capability may be stronger. Andrew says it is fair to penalize overly strict safety classifiers; Astra scored 0% on cybersecurity because it refused the tasks. Muse Wonder and Muse Wonderful emerged as “top tier” cyber options, reinforcing Jordan’s experience, although Wonderful was not available through the public API.
- Carbon improves Google’s competitive position without closing the leading labs’ gap in Jordan’s preferred comparison. Excluding biology, cybersecurity and alignment, and using partial grading, Gemini 4 Carbon still trails Astra and Opus. Jordan sees Carbon as an inflection for Google, calling it the “third place player”; he also notes that new OpenAI and Anthropic flagships are expected soon.
- The employment debate remains unresolved, with sharper pressure on junior work than a simple productivity story implies. Max sees human judgment making good engineers more valuable today, while allowing that far fewer supervisors might eventually be needed. Andrew challenges the assumption that displaced workers will become more ambitious: many people simply want to “put their hours in and go home.”
Deep dive
1. A difficult benchmark must first be a fair benchmark
Max’s core distinction is “difficult versus unfair”: a model can correctly fix a coding bug using module B, while a verifier accepts only module A. The failure then reflects the grader’s mismatch with the prompt. He argues that widely used benchmarks contain many such problems; identifying them requires examining rollouts and judging why the model failed, rather than simply inspecting the task description. Andrew adds that models may solve tasks in unexpected ways, so the team reviewed transcripts and sometimes used AI-assisted probing to assess whether a result was correct and fair.
SemiAnalysis did not assume it could invent 200 excellent tasks itself. It recruited 14 specialized data providers, handled QA centrally, and rejected more providers than it ultimately included. Accepted partners still underwent substantial iteration. Max’s point is that making tasks simultaneously realistic, fair and sufficiently difficult requires domain expertise as well as repeated examination of actual model behavior.
The benchmark supplies a repeatable way to assess releases the team cannot personally use extensively. Its final ranking was genuinely unfamiliar: tasks were finalized Thursday for a Sunday launch, and the complete leaderboard arrived only three to four hours before publication. Execution plus grading could easily take 24 hours. Carbon was among the models the team had scarcely encountered before evaluating it; they had access on Thursday night and were trying it for the first time.
2. Frontier revenue depends on work that keeps rewarding intelligence
Max’s guiding thesis is that “the first trillion dollars of AI revenue” will concentrate in domains capable of absorbing increasingly intelligent models at high ROI. He agrees that most tasks will not need frontier intelligence, and expects that to be especially true heading into 2027. GDPX therefore targets the frontier providers’ prospective revenue engines rather than a representative cross-section of all knowledge work.
The dashboard lists recursive self-improvement, chip design, cybersecurity, biology, finance, legal, quant trading, hardware design, materials science, alignment and software engineering. Selection emphasized verifiability and capacity to absorb intelligence. Alignment also reflects a broader evaluation priority: the team considers it essential for future models, with Fleet’s expertise making its inclusion practical.
The team’s account of materials scientist Brian captures a change of mind. He initially did not expect Astra to be so good at some of the workflows he proposed. Once he made tasks, he realized he had been “prompting it wrong” or supplying insufficient context, and needed a harder task set. Max says Brian watched Astra do work that would place a human materials scientist above the median.
Max says the usual human-baseline comparison becomes difficult when Astra beats a published state-of-the-art reference on a materials-science subtask, and some chip-design tasks would take teams at companies. Jordan supplies the economic contrast: the largest materials-science tasks cost $40–$50 each, and some models solved tasks for around $10.
3. Safety filters and specialist strengths reshape the leaderboard
The headline ranking includes biology, cybersecurity and alignment, although safety classifiers sometimes suppress performance. Astra’s cybersecurity score was 0% because it refused the task. Andrew says it is fair to penalize overly strict safety classifiers to some degree.
Jordan independently found Muse Wonder strong on cyber; Muse Wonderful’s result reinforced that experience. The team obtained Wonderful despite its absence from the public API. “You guys gave away all my alpha there,” Jordan jokes. He emphasizes filtering results around the work and access a user actually has. Andrew says the team had only the bottom or perhaps second-to-worst tier of Anthropic access, which still triggered restrictions on many tasks.
Andrew says he was glad his subjective experience with Opus 5.5 was validated, while acknowledging that Astra was exceptionally strong in materials science and some engineering work. Different labs have different strengths. His practical takeaway is to use whichever model performs best for the work at hand, rather than assume one model wins everywhere.
Jordan’s preferred apples-to-apples comparison excludes biology, cybersecurity and alignment, and uses partial grading for finance and legal. Under that comparison, Gemini 4 Carbon remains behind Astra and Opus. Jordan calls Carbon an inflection for Google and says it has reestablished itself as the “third-place player,” while also describing Google as at the top of the Tier 2 cohort. He says the gap with OpenAI and Anthropic remains meaningful and notes their flagship releases are expected soon. Max adds that the comparison is between public models; OpenAI and Anthropic may not be releasing their best models.
4. A real acquisition exposes finance’s context problem
Max contrasts asking a banker to invent a realistic task in two hours with reconstructing an actual transaction. GDPX’s finance partner used a private-equity acquisition of a family-owned snack company: a little over $800 million bought 62%. The environment contained all 4,000 real files. For Max, this is “categorically different” from asking an expert to create a synthetic approximation of their work.
The files span the private-equity firm, investment bank, accounting firm and seller’s data rooms, preserving their actual organization and disorder. Tasks came from work people really performed, including instructions in emails from accounting partners to associates. Jordan particularly values the ability to replay the data room as it existed when a task was assigned, rather than give every task access to the final document collection.
One failure involved an email instructing someone to adjust figures to Q4. Missing that message meant producing incorrect numbers despite otherwise plausible work. Jordan says he checked with finance friends, who told him an associate would absolutely be expected to catch it. In Jordan’s words, the same omission would be “unacceptable behavior in a real banking environment.”
5. Partial credit shows progress without making a workbook usable
Finance tasks receive binary final grades, although the workbook’s components are scored individually. Max’s rationale is operational: a quality-of-earnings workbook with half its numbers wrong is not a satisfactory deliverable. A supervisor will not say, “great job, little Jimmy,” and finish it themselves. Partial scoring describes competence; the final verdict reflects whether the assigned work was completed successfully.
Binary finance results were essentially zero, apart from Carbon, which got one task right. Jordan says Opus’s partial score was in the 50-something-percent range. Legal remained all zeros under binary grading; Jordan says the best partial score was in the 30s or 40s. These gaps matter because they indicate substantial missing capability rather than one marginal requirement blocking otherwise complete work.
Max accepts the criticism that binary grading can disguise easy or unfair tasks: a 25% binary score alongside 97% partial might mean the remaining 3% is an esoteric requirement the model should have been expected to meet. Jordan adds that a large gap might instead indicate an unfair or broken task. GDPX uses continuous grading in most domains. Chip-design PPA optimization, for example, can score progress between a naive baseline at zero and a theoretical maximum at one.
Finance’s prospective token spending rests on margins that can support substantial costs; Max says it is already a major spending domain in 2026 and expects heavy spending in 2027. Legal adds a “PvP” mechanism: if better AI could improve litigation outcomes or contract negotiations, an opponent using it could create pressure to buy stronger intelligence too.
6. Better models strengthen some workers while threatening junior roles
Andrew once believed humans plus AI would remain much better and that he would keep writing code manually. He describes Sonnet 3.7, Sonnet 4, Sonnet 5, and especially Opus 4.5 and 4.6 changing his view. He still thinks humans plus AI outperform AI alone because people contribute “little pieces of entropy” the models do not possess themselves.
Max separates benchmark results from a job-loss forecast: GDPX deliberately omits vulnerable work such as customer service. Good engineers currently add “taste and supervision and judgment,” but maybe in two years one supervisor can replace most of a company’s engineers; maybe not. Intern assignments are already harder to justify when models handle well-specified problems and managing an intern adds overhead.
Jordan argues that capability can expand demand: cybersecurity needs more expertise, and materials discovery can generate additional work. Max counters that a “cracked” materials scientist might nevertheless automate a PhD assistant’s job. An unidentified speaker in the transcript offers the optimistic possibility that a former grunt worker could use “infinite PhD grunt workers on tap” to pursue more ambitious science.
Andrew explicitly disagrees with that optimistic view as a general expectation: many Americans, he argues, want to work their hours and go home rather than continually elevate their careers. Max suggests AI may force people to care more because otherwise they will not provide value, but the outcome remains unresolved. The disagreement concerns human adaptation, not whether the tools are improving.
7. Keeping tasks private protects the benchmark’s purpose
Jordan describes the disclosure trade-off: publishing tasks lets outside experts inspect quality, but also makes hill-climbing easier. A benchmark can stop measuring capability and start measuring which laboratory bought the most similarly shaped training data. GDPX’s first priority is preserving a measure of “real model capabilities,” rather than maximizing public access to every task.
The compromise is one released task and rollout per partner, supported by detailed methodology. Evidence includes the real finance deal room and cybersecurity partners turning their own zero-days into tasks. Max says the team ran its own agent for QA; he cites Bench 4 flagging about 50% of its tasks as broken, and says GDPX tasks were then run by every partner, with tasks that failed the bar excluded. The team seeks credibility through concrete construction details and sample behavior while keeping most evaluation material private.
The next ambition is GDPX v1 appearing on major model-release cards, followed by versions that respond to task saturation and newly valuable domains. Max and Andrew want more strong partners, more tasks per domain and evaluations across every reasoning level. The intended benchmark evolves with the work frontier models can perform, rather than freezing today’s task mix indefinitely.