Summary

Cloudflare’s current pay-per-crawl structure will not revive incumbent publishers. Ben agrees with Finn that AI companies can index or cache pages instead of repeatedly paying to crawl them; he would not enable the system and sees no “meaningful revenue stream.” He says the proposal is directionally pointing at a real need for content markets, but making one work would require someone with enough power to impose it. At most, it may facilitate brand-oriented licensing deals for large publishers such as The New York Times—“the folks that kinda need the least help.”
Publishers face a structural leverage problem: AI needs content, but it does not need any particular publisher’s content. Ben compares the imbalance to Google needing the web while individual sites need Google, a dynamic also undermining copyright plaintiffs. Andrew adds that search referrals are falling, the advertising model has deteriorated for 15 years, LLMs are becoming the portal, and ambitious media cost structures already fail; expecting licensing to “make it 1997 again” is a “fantasy.”
If AI does create sustained demand for more material, the winners will probably be new suppliers built specifically for that market. Ben’s model is “ghost kitchens for content”: just as delivery-only restaurants optimize every workflow for food consumed 30 minutes later, AI-native producers would optimize for machine ingestion. The market comes first and summons its suppliers, as Apple’s App Store, Meta advertising, and Amazon’s seller ads did.
The resulting labor market would be a barbell, not a broad revival of professional media. At one end, prestige creators and institutions retain direct-paying audiences; at the other are outsourced “salt mines of content” producing fungible inputs for models. “The LLMs, they don’t even want content. They want tokens.”
The New York Times is the Taylor Swift of publishing—proof that an exceptional model works, not a template for the median outlet. Its advantage came from aligning editorial choices with subscription value, bundling early, and making work readers would “feel good about paying” for; its dominance can leave everyone else competing to be the second subscription. Ben’s harsher diagnosis is that AI is “accelerating the death” already underway and might amount to a “mercy killing.”
A Spotify-like pool could eventually price machine-consumed material, but it would price a commodity rather than preserve prose economics. Payments might be allocated by measured share of usage, avoiding uncapped per-crawl liabilities, though analytics would be difficult. Andrew notes that songs remain distinct while facts—such as who won a Wizards game—are available everywhere; Ben’s conclusion is that the market would be “mining tokens.”

Deep dive

1. Pay per crawl fails because models can index around the toll

Finn’s challenge was concrete: training may require one pull, perhaps another six months later for “Llama 3.3-500B_26-04,” while inference traffic can be routed through a Redis cache and crawled at a cost-efficient minimum. His taunt to Ben: “Congrats, Ben. You made four cents last week.”
Ben agrees that Cloudflare’s current structure “is not going to scale” and does not see it becoming meaningful publisher revenue. He says the idea is directionally pointing at something real, but getting there would require someone with power to make the system happen. AI systems build indexes; he would not turn pay per crawl on himself.
The deeper mismatch is bargaining power. As Ben puts it, Google needs the web but not any one website, while each website needs Google; similarly, AI models need abundant material but “don’t necessarily need your content.” That weakness also dogs copyright plaintiffs, including those in cases before Judge Chhabria.
Andrew’s incumbent-media ledger compounds the problem: Google sends less external traffic, advertising has been deteriorating for 15 years, audiences are diminished, LLMs may replace site visits, and quality publishing carries a cost structure that no longer makes sense. Licensing is unlikely to restore 1997 economics; Andrew calls that prospect a “fantasy.”

2. New markets can manufacture their own suppliers

Ben allows that models might already be sufficiently capable, needing only something like one wire service for current events. If they require substantially more material, however, some market must arise to induce its production. Andrew also raises synthetic data as part of the possibility.
Amazon’s multibillion-dollar advertising operation supplies the pattern: its advertisers are sellers operating on Amazon itself. Meta similarly created businesses dependent on performance marketing, making them resistant to a CPG boycott or an early-COVID pullback—those companies must “advertise or die.”
Google is the exception because the open web already existed, waiting to be captured; Ben says the company “played on easy mode” and has harvested ever since. Search advertising can resemble a tax when a sponsored result intercepts a click that would have gone to the organic link immediately beneath it.
Ben envisions a possible AI equivalent in “ghost kitchens for content.” DoorDash and Uber Eats helped create a market for delivery, including virtual restaurants whose costs, workflows, and food are structured around consumption roughly 30 minutes later. An AI-content market could likewise attract producers designed for machine demand rather than retrofit legacy newsrooms.

3. Demand comes before developers—and creators

Ben’s App Store lesson, informed by his work on the Windows 8 App Store, is unequivocal: “It’s absolutely the market comes first.” Apple first built a phone people wanted, then gave developers a mechanism to serve that installed demand; most suppliers arrived because the market existed.
Future machine-oriented producers may therefore publish exclusively for crawlers, without maintaining consumer websites. Ben cautions that these would not necessarily be attractive jobs, pointing to Scale AI’s outsourcing of labor to the Philippines or elsewhere.
His “barbell effect” leaves prestige brands with direct consumers at one extreme and commodity labor at the other—the “mines, the salt mines of content.” Machine customers are not buying authorship or presentation; they seek statistical associations among tokens.

4. Subscription winners cannot rescue the publishing middle

Paul’s music analogy runs from CDs through piracy and streaming to creators finding an “Eras Tour.” Ben accepts the bifurcation but warns that Taylor Swift is singular. Andrew adds that live shows have worked as a business for a broader cohort than Taylor Swift alone; The New York Times occupies the analogous exceptional position in publishing.
The Times solved its problem through subscriptions rather than advertising, aligned its editorial mission with being worth paying for, and moved early into bundling. Its scale also hurts rivals: it attracts top talent and leaves other outlets competing to become a customer’s second subscription.
Ben’s emblematic example is the Times exposé of Amazon working conditions. Whatever one’s politics, the newsroom saw that ambitious, provocative work gave readers a reason to subscribe—“the stuff people will pay for”—and favored it over routine day-to-day coverage.
That integration of business model and editorial mission replaced a firewall Ben regards as a vestige of geographic monopoly. The Times won by aligning the whole operation; everyone else was already deteriorating, and AI is “accelerating the death,” not originating it.

5. A pooled market would value tokens, not publishers

Ben can imagine a Spotify-like market maker: collect a pool of money and distribute it according to each supplier’s measured share of usage, just as Spotify allocates music revenue by share of plays rather than promising a fixed penny per stream.
The unresolved problem is analytics and measurement—how to determine usage and allocate credit. Pooling would constrain the total amount paid rather than create an uncapped liability, but it would also formalize content as a commodity, like Ben’s analogy to the amount of money available for iron ore or oil. Anyone seeking unlimited pricing power would have to operate outside the pool.
Andrew sharpens the distinction from music: one song differs from another, while a fact such as who won a Wizards game can come from many places. Ben agrees that publishers possess even less leverage than musicians. The prospective industry is not really selling content at all: “That market is mining tokens.”

Trump Fights ‘Woke’ A.I. + We Hear Out Our Critics No Priors Ep. 124 | With SurgeAI Founder and CEO Edwin Chen