Summary

Modal’s core wager is that scarce AI compute becomes a pooled, usage-based utility: thousands of GPUs and CPUs, with 100 GPUs typically available within seconds. Bernhardsson argues that startups should not need long-term commitments for volatile inference demand; pooling workloads across customers converts idle-capacity risk and service degradation into a capacity-management problem Modal absorbs.
The product advantage comes from building a cloud-native substrate rather than relying on Docker and Kubernetes alone. To make cloud development feel “almost as good as local development,” Modal spent its first two years building a file system, scheduler, and container runtime that can safely execute custom Python while minimizing cold starts.
Inference was Modal’s accidental killer app, but the expansion thesis spans preprocessing, short experimental training runs, inference, and feedback pipelines. Stable Diffusion triggered adoption, while customers now process petabytes of video on Modal, train elsewhere, and return for inference; Bernhardsson says very large training runs are “a very different market.”
The strategic tension is between Modal’s multi-tenant efficiency and enterprises already anchored to AWS, Azure, or GCP through data gravity, credits, latency requirements, and security reviews. Bernhardsson concedes the challenge but sees the same adoption arc as cloud and Snowflake: security moving from the network to the application layer, bandwidth costs declining, and options such as R2 offering zero egress fees.
For companies whose differentiation depends on model quality, Bernhardsson sees a clearly superior proprietary model as an inherent technical moat. A company using the same off-the-shelf model as everyone else must find defensibility elsewhere; for audio, video, and image products, his view is sharper: “you kind of have to train your own models.”
AI coding may expand software employment because each productivity step unlocks more latent demand rather than satisfying a fixed market. Bernhardsson compares AI with compilers, higher-level languages, databases, and cloud infrastructure: over roughly 40 years, engineers became “10 times more productive” each decade while demand kept rising.
AI-native opportunities may emerge beyond conventional model serving, particularly in storage, scientific computing, and generative media. Bernhardsson thinks it may take 5–10 years for AI-native storage to shake out, sees computational biology and medical imaging gaining traction on Modal, and calls Suno a product that “couldn’t have existed five years ago.”

Deep dive

1. Modal rebuilt the cloud stack to make remote compute feel local

Bernhardsson traces Modal to infrastructure work beginning at Spotify in 2008, where he spent seven years building a music recommendation system amid an immature data stack. His candid inventory includes Luigi, “that basically no one uses today,” and Annoy, a vector database that people used “for a brief period.”
After serving as Better.com’s CTO and thinking deeply about developer productivity, he returned during the pandemic to a recurring problem: cloud infrastructure was powerful but “actually pretty frustrating to work with.” The product question became, “What if you could make cloud development feel almost as good as local development?”—especially its fast feedback loops.
That ambition ruled out treating Docker and Kubernetes as sufficient foundations. Modal spent roughly its first two years building its own file system, scheduler, and container runtime, creating the machinery needed to boot containers quickly and turn ordinary Python functions into serverless cloud workloads.

2. Pooled GPUs turn unpredictable inference into a utility

Modal operates one multi-tenant pool containing thousands of GPUs and CPUs; Bernhardsson says a customer needing 100 GPUs can typically obtain them within seconds. The economic proposition is capacity planning as a service: customers avoid reservations and pay only while their containers are actually running.
Bernhardsson’s objection to prevailing GPU procurement is structural. The cloud’s original promise was instantaneous access to whatever compute a developer needed, yet GPU scarcity pushed startups toward long-term commitments—“fundamentally…just not how startups should do it.”
Gil observes that training teams can assemble a giant cluster, run it for months, and—as he puts it—produce “a file” representing an internet-scale corpus. Bernhardsson contrasts this with inference, where demand is volatile and operators either over-provision and pay for idle GPUs or under-provision and suffer degraded service.
Modal historically concentrated on inference, initially propelled by Stable Diffusion and later by image, video, audio, and music workloads. Bernhardsson is pursuing “shorter, very bursty” experimental training runs that benefit from elastic capacity, while saying very large training runs are “a very different market.”

3. The platform thesis stretches across the ML lifecycle

Bernhardsson’s intended customer is the “high code” ML engineer building custom models and workflows. The eventual platform spans data preprocessing, training, inference, feedback loops, online ranking, data pipelines, and nightly batch jobs—not merely a hosted endpoint for language models.
Existing usage already foreshadows that expansion: a customer training a video model might process petabytes of footage on Modal, use GPUs for feature extraction, train elsewhere, then return for inference. Modal does not yet have a training product, but Bernhardsson says adding one “makes a lot of sense.”
The differentiation claim rests on two choices: being “cloud maximalists” with shared multi-tenant capacity, and safely running custom customer code. Competitors may specialize in inference or LLMs; Modal accepts the harder cold-start, isolation, scheduling, and containerization problems to remain general purpose.

4. Multi-tenancy must overcome enterprise data gravity

Gil’s pushback is that enterprises already store data on AWS, Azure, or GCP, possess marketplace credits, and have completed security reviews. Sending workloads to another provider can introduce latency, bandwidth, and compliance friction, particularly for organizations operating across multiple hyperscalers.
Bernhardsson concedes this is “a challenge” but compares the skepticism with his own reaction to cloud computing in 2008–2009: “Why would anyone put their compute in someone else’s computer?” He similarly thought Snowflake’s infrastructure-as-a-service approach around 2012 was “nuts,” before Snowflake became a very large, publicly traded company.
His rebuttal is directional rather than absolute: he sees security shifting from the network layer toward the application layer, bandwidth costs declining, transfer being minimized, and R2 offering zero egress fees. Those tailwinds matter because he argues that dynamically slicing a large compute pool creates substantial capacity advantages.

5. Models, storage, and software demand remain open frontiers

Bernhardsson thinks demand has shifted somewhat toward what he calls “proprietary open source models,” with Flux receiving attention. Audio remains “very underexplored,” although he says, “I don’t think we’ve seen anything really cool yet.”
On vector databases versus Postgres with pgvector, his honest answer is, “I don’t necessarily know.” The more consequential question is whether an AI-native store should embed text, images, and video itself, rather than accept precomputed vectors through a conventional put-and-get database interface; he thinks it may take 5–10 years for this to shake out.
Bernhardsson also wonders whether less bandwidth-hungry training could connect GPUs across data centers, reducing dependence on enormous InfiniBand-equipped facilities. Separately, Modal is seeing computational-biology demand, including computer vision over millions of automated electron-microscope images, though he carefully qualifies his account as “kind of superficial.”
Gil explores deep learning for simulation, turbulence, weather, and HPC; Bernhardsson sees meteorology as a promising application in theory but does not endorse a broader physics renaissance outright. His clearest human-impact example is Suno: AI music still has a sort of “uncanny valley,” yet each generation improves, and the product represents something generative AI made newly possible.
On coding, Bernhardsson rejects a fixed-demand framing. Compilers, higher-level languages, databases, and cloud tooling repeatedly multiplied productivity, yet “there’s so much latent demand for software” that the number of software engineers went up; AI may be another step in that long pattern rather than an exception.

Tyler Cowen — The #1 bottleneck to AI progress is humans Francois Chollet - ARC reflections - NeurIPS 2024