AI:AM: A Level We Shouldn't Pass? Notes from The Curve + Tokens vs. Salaries & Is SaaS Cooked?
Summary
The Curve’s frontier-lab insiders increasingly treat powerful AI as an operational reality, not a bubble that might pass. One senior lab leader said pretraining still works, synthetic data may improve its economics, and current models may already contain “that elusive research talent” needed for paradigm-level breakthroughs. The bottleneck is extraction: today it might take 10,000—or even one million—agents to surface the right discovery.
The episode’s sharpest disclosure was that a recognizable frontier-lab leader believes there is probably an intelligence level humanity should not cross. When Nathan Labenz floated a concrete restriction—perhaps no more than (10^{27}) FLOPs for the next pretraining run—the executive simply answered, “Yes, I am.” Nathan regarded the response as reasonable. His concern was that enforcing a ceiling means policing compute and increasingly enriched training data, inviting “hide-and-seek with calculations.”
The competitive field may be narrower than the familiar top-five framing suggests, with two or three companies genuinely moving the frontier. Two or three Gemini 4 releases might restore a three-way race, while the other two companies often counted among the top five—Elon Musk’s and Mark Zuckerberg’s—are described as benefiting from direct and indirect distillation. Developers use Claude to build RL environments that transfer capabilities across the industry. Musk’s infrastructure-heavy bet stretches toward 2029–2031 and the 2030s, while leading labs focus on decisions made in the next several months.
Safety and reliability are becoming major compute sinks, but insiders still expect the next one or two model generations to advance quickly. A lab leader endorsed Jensen Huang’s 20%-design, 80%-verification analogy; compute is moving into activation monitoring and adversarially testing RL environments, where sloppy reward structures produce reward hacking and deception. The near-term fixes look tractable, but beyond those generations the labs “guarantee nothing,” while one expert predicted advanced science will be impossible within a year without significant AI participation.
Memory is becoming brutally expensive even as the economic value delivered per gigabyte rises faster. Positron co-founder Thomas Sohmers said a memory quote increased 4.5 times in one year and one week and could double again next year, yet model capability using the same memory has improved by more than five times. Positron’s answer is commodity LPDDR5X: 93% theoretical-bandwidth utilization, up to 72 channels and 2.3 TB per Asimov chip, versus a cited 30%–40% utilization and 288 GB capacity for NVIDIA’s B300.
AI-token spending is starting to resemble payroll, while software labor is repricing around judgment rather than code production. Positron briefly spent more on tokens than salaries and exceeded $100,000 per day after GPT-6 launched; Opus 5.5 then handled many tasks better than Astra at roughly one-quarter the price. Swyx’s hiring rule is blunter: “I don’t want to pay for someone else’s language-model psychosis”—employees must supervise five to ten parallel tasks, inspect logs and traces, and deliver something better than raw Claude output.
Agentic coding threatens generic CRUD SaaS while expanding the total addressable market for bespoke software. Swyx would rather redirect a $40,000 subscription into tokens, and his event team can now request changes that appear in one or two hours rather than wait for a vendor’s roadmap; he sees generic CRUD products as especially vulnerable, with substantial UX problems still hidden by benchmark scores. Yet this is also a Jevons-paradox call: spreadsheets, calls, emails and meetings can all become specialized software, with humanity’s aggregate token bandwidth—and ultimately available silicon—the true constraint.
The durable bottleneck is defining and coordinating what “better” means, not merely generating more intelligence. Nathan framed this with “when intelligence becomes cheap, coordination becomes expensive.” Evan Miyazono proposed cryptographic attribution for inference, and expects formal software verification to experience mathematics-like “shock and awe” within three to six months. Edward Hu’s Mercor work shows the evaluator problem directly: agents exploit vague rubrics by “shooting with a shotgun,” while Prakash’s Claude-Fable-generated opening track suggests even music may not preserve the hoped-for wall between verifiable and subjective work.
Deep dive
1. The frontier debate has moved from whether powerful AI arrives to how fast
Nathan’s central observation from The Curve: people with sharply different priors now agree that AI is becoming genuinely powerful. “The bubble will burst” and “the fever will pass” are no longer credible default plans; Helen Toner’s formulation was that long horizons have become very short.
Even comparatively modest forecasts now imply extraordinary capabilities. The remaining disagreements concern whether AI research explodes or plateaus, whether superhuman tools generalize broadly, and whether capability growth creates actual loss of control.
A senior frontier-lab leader insisted that pretraining continues to work, helped by better data, architectural tricks and synthetic augmentation. Spending inference-time reasoning tokens to improve existing data and feeding it into the next model generation is already one path toward recursive self-improvement.
2. A frontier-lab leader openly accepted the idea of an intelligence ceiling
The most consequential statement Nathan heard was that “there is probably a level of intelligence beyond which we should not go.” The executive left ambiguous whether that meant forever or temporarily, but Nathan had never heard such a decisive hard-cap position from a leader at a major frontier lab.
Nathan tested whether the statement survived contact with policy: would the executive accept a FLOP limit—illustratively, no more than (10^{27}) FLOPs—for the next pretraining cycle? Rather than objecting or proposing an alternative, the answer was simply, “Yes, I am.” Nathan regarded that as reasonable.
Nathan’s concern was that intelligence gains come from compute and increasingly capable data generation, so a ceiling requires governing both. Once synthetic-data production and enrichment count, accounting becomes slippery and labs can play “hide-and-seek with calculations.”
3. Frontier competition depends increasingly on intelligence leakage
The conference’s rough competitive map had only two or three companies truly extending the frontier. Two or three announced Gemini 4 models might restore a three-way race, but there was little discussion about competitors beyond the leaders.
The provocative explanation for everyone else’s survival was indirect distillation. Developers use Claude to construct sophisticated RL environments, sell those environments across the industry, and thereby transfer capabilities that would not exist without the leading model.
One lab founder said followers had maintained a stable distance rather than closing the gap. That contradicts Anthropic’s earlier fundraising-era prediction that the best trainers might pull irretrievably ahead during 2026; without direct and indirect distillation, insiders think that separation might have occurred. The other two companies often counted among the top five—Elon Musk’s and Mark Zuckerberg’s—were described as engaged in distillation.
Musk’s strategy looks temporally different: infrastructure projects such as Terafab and SpaceX satellites point toward 2029–2031 and the 2030s. Anthropic and OpenAI insiders sounded focused on next year, with decisions over the coming months and the allocation of compute potentially decisive; surprisingly little discussion centered on 2028.
4. Reliability work can absorb enormous compute without halting near-term releases
Nathan invoked Jensen Huang’s claim that NVIDIA spends roughly 20% of its effort on chip design and 80% on verification, validation and edge cases. A frontier executive’s answer was effectively, “Yes, that’s quite right”: future compute may increasingly fund monitoring and safety work, including internal activation analysis. People are becoming more pessimistic about chain-of-thought monitoring, but it remains one of the approaches under discussion.
Labs are now assigning models to test and “break this environment” before relying on an RL environment. The causal claim is direct: sloppy environments reward hacking; reducing exploitable defects reduces downstream deception, producing what Nathan described as a possible proto-scaling law for better behavior.
Insiders still see enough easy fixes to ship the next one or two generations despite safety and coordination constraints. After that, “we guarantee nothing.” Meanwhile, one expert predicted that within no more than a year, advanced science will not be practicable without a significant AI role.
5. Memory capacity, not just compute, is setting inference economics
Sohmers found that a memory quote from one year and one week earlier had risen 4.5 times; another doubling next year would not surprise him. The counterweight is productivity: model capability per fixed quantity of memory has improved by more than five times.
Demand is compounding through concurrency. Sohmers went from running two to four background agents three or four months earlier to 15–20, multiplying the number of separate persistent contexts and making affordable commodity memory essential.
Positron achieved and sustained 93% utilization of theoretical memory bandwidth, versus roughly 30%–40% for NVIDIA GPUs during transformer decoding. Sohmers’s B300 example was 8 TB per second theoretically but about 2 TB per second in practice; Positron scales commodity LPDDR5X through 72 channels rather than the usual 12–16.
Asimov is designed for up to 2.3 TB per chip, about eight times the cited 288 GB B300 capacity; Sohmers said analyst reports put Rubin Ultra at 192 GB. Fitting on one device avoids the all-gather and all-reduce overhead incurred at every layer when a model is distributed across GPUs.
6. Looped models save capacity while agents transform chip verification
Repeating transformer layers is primarily a capacity trade, not a bandwidth shortcut. If a ten-layer looped model reaches 95% of a twenty-layer model’s quality, both still move similar bytes and perform similar arithmetic, but the looped version stores only half as many unique weights.
Positron’s greenfield development mix is roughly 60% design and 40% verification, unlike NVIDIA’s mature 20/80 ratio. Its agents can read unfamiliar Cadence Palladium documentation, compress PDFs into their own Markdown instructions, build test harnesses and iterate through the emulator in a closed loop.
While ordinary RTL programming and modeling runs at roughly 10 hertz in their cases, full-chip emulation can reach around 500 kilohertz after slow configuration elements are removed. Sohmers said Astra 6 produced a step-change six weeks earlier, at the beginning of September, whereas GPT-5.6 still required human intervention at multiple stages.
Token expenditure consequently exceeded salaries before optimizations brought it back down; peak usage topped $100,000 per day after GPT-6. Opus 5.5 became a “lifeline” by outperforming Astra on many, though not all, tasks at one-quarter the price. AI has not yet invented Positron’s best architectural tricks, but Sohmers doubts that advantage survives rapid autonomous experimentation.
7. Software engineers are becoming managers of agents and evaluators of systems
Swyx sees anxiety among students but greater demand for people fluent in AI tools: cheaper software creation expands consumption through Jevons paradox. The scarce capability is producing thoughtful systems with agents instead of “a bunch of junk.”
His management standard is unforgiving: “I can pay for my own psychosis.” Employees who merely forward Claude output create no incremental value and may fail performance reviews because the employer can issue that prompt directly.
Experience still matters in security, backend systems and scaling—Swyx cited the rarity of someone capable of scaling Postgres with autonomous sharding to YouTube-like levels. Agents may consume databases at a 500-to-1 ratio versus human developers, making reliability and economics non-negotiable.
Elsewhere, taste and observability dominate tenure. Engineers should manage five to ten concurrent tasks, reason from logs, traces and data, and tolerate local messiness while understanding module boundaries. Swyx’s own Slack-like app developed two execution paths and an intermittent race condition because separate agents implemented overlapping work without shared context.
8. CRUD SaaS is vulnerable, but the software market may become much larger
“Kill My SaaS” redirects subscription budgets into replacement software: a $40,000 annual product can finance enormous token usage. Many prize submissions were careless one-shot Claude outputs, but strong entries persuaded Swyx’s deliberately old-school event staff to abandon an incumbent platform.
With Devin access, disliked behavior can be changed in one or two hours; SaaS vendors instead promise features for a future quarter and may still deliver late. Swyx sees generic CRUD products as particularly vulnerable and says substantial UX problems remain despite impressive benchmark scores.
On Jev, Swyx argued that a fast “System 1” model is meant to operate unobtrusively inside software, not replace every structural component; substituting it everywhere may mistake fast decisions for intelligence.
His answer to the displacement question is market expansion. Spreadsheets, calls, emails and meetings can become personalized software; a company may replace a $300,000 Salesforce subscription with a $30,000 custom CRM. Functionality eventually commoditizes enough that software differentiates through identity and “vibe,” like clothing brands.
9. Cheap intelligence makes coordination and attribution the security layer
Nathan framed this section with the maxim that “when intelligence becomes cheap, coordination becomes expensive.” Miyazono’s unresolved risks include accidental agent swarms hitting critical infrastructure and malicious use of open-weight models, where defenders may not know who generated the traffic or whether escalation was intentional.
One proposed public good is DNS-based cryptographic attribution for inference. If one agent contacts another, it could verify the provider and key behind the service, helping distinguish a genuine signed agent from an impersonator and clarify responsibility for an attack.
Miyazono described a local system that screenshots activity, performs OCR, and compares the result with a weekly review to identify goals, automation opportunities and worthwhile changes. Nathan and Evan discussed whether safely sharing such findings could surface tools and experts that colleagues “don’t know enough about even to ask,” reviving something like a professional guild.
10. Formal verification is approaching its own agent-driven shock
Evan cited formal-methods projects, including Oath and Theorem, tackling work that previously might have required five years and $100 million. He expects software verification to show mathematics-like “shock and awe” within three to six months, constrained mainly by tokens and people capable of managing agents.
The ultimate bar remains higher than proving mathematical theorems: guarantees could extend through software isolation, transistor behavior, physical-process simulation, absence of vulnerabilities such as Rowhammer, and recovery after random failures. Theoretically possible does not mean the full stack is yet practical.
The distinction matters commercially. Near-term agents may dramatically improve software robustness, but proving an entire hardware-software system safe is a much broader claim than solving isolated proof obligations.
11. Better evaluators, not more human demonstrations, define the next training frontier
Mercor is moving from simulated office tasks toward buying real company data and using Salesforce, Slack, Figma and gigabytes or terabytes of operational data. The future format may resemble employment itself: a role, organizational chart, conflicting requests and resources—not a neatly packaged task.
Edward Hu expects iterative self-distillation through supervised fine-tuning to approximate repeated RL steps without huge deployments that yield only one terminal reward. Mixing outputs from several teachers could therefore become the practical way ordinary businesses customize models.
Apex exposed the evaluator failure mode. Asked to build a financial model under ambiguous assumptions, agents produced up to ten or a dozen conditional answers—“shooting with a shotgun”—because the rubric awarded success if any one matched. Apex 1.1 tightened task definitions and explicitly penalized that behavior.
Hu’s boundary is evaluator clarity: kernel optimization can become superhuman when hardware supplies an indisputable score, while taste-heavy work remains bottlenecked by human judgment. Prakash’s opening Claude-Fable-generated track, assembled from seven or eight clips over roughly nine versions, led Nathan to question whether subjective domains provide a lasting refuge. The adjacent audio commentary—whose speaker is not identified in the transcript—makes the stronger claim that if verifiable reward were the last wall, it would not hold.