Prof. Randall Balestriero - LLMs without pretraining and SSL
Summary
For narrow perception tasks, a randomly initialized 7-billion-parameter transformer can learn from roughly 20,000 labeled samples and sometimes match a pretrained model with LoRA fine-tuning. Training was stable—“almost like you train MNIST”—despite an extraordinary parameter-to-sample ratio, challenging the assumption that expensive pretraining is always necessary for specialized classifiers.
The economic implication is conditional, not a blanket verdict against pretraining. Balestriero says next-token pretraining remains necessary for generation and valuable for open-ended reasoning or distribution shifts; the cheaper model is highly specialized and cannot answer questions absent from its supervised data. For fixed, in-distribution classification, however, pretraining was “not better than random initialization.”
Specialized supervised objectives may avoid much of next-token prediction’s computational waste. “The FAIR Language Model Paradox” attributes long training partly to low-frequency tokens that a generative model must capture even when they are irrelevant downstream. In these experiments, ordinary supervised training required no hyperparameter optimization and was used “out of the box,” suggesting a materially simpler path to deployment.
The real design space runs from one-task specialists to universal next-token models, with multitask training potentially capturing much of the useful middle. Balestriero proposes finding the minimum set—perhaps five diverse tasks—that yields reusable representations without reconstructing full next-token prediction. The commercially relevant choice becomes application-specific: expected task drift determines where to sit on the spectrum.
Balestriero argues that tokens are unnecessarily granular units for machine “thinking.” Predicting the exact duration in “I saw this movie for ___ minutes” is less important than representing the concept of elapsed time; “concept tokens” might provide a more meaningful abstraction while avoiding lossless reconstruction of raw language.
Self-supervised and supervised learning are mathematically connected through how samples are related, not fundamentally separated by their losses. Least-squares supervision can map to VCREG, VICReg, or W-MSE, while cross-entropy maps toward SimCLR-like objectives. This correspondence lets researchers transfer mature supervised-learning theory—including imbalance correction and neural-collapse results—into SSL.
Earth-data models can look strong globally while becoming “almost random” around islands and coastlines, creating direct policy and allocation risk. Fourier bases impose stationarity that poorly fits localized gradients; wavelets reduce some bias but are not a universal solution. Balestriero’s broader warning is that average accuracy conceals geographic failure modes, especially when crowdsourced data follows human population density.
Deep dive
1. Random initialization is competitive when the task stays narrow
Balestriero’s motivating test compared a pretrained 7-billion-parameter model, lightly adapted with LoRA, against a randomly initialized model for sentiment and job-description classification. With about 7 billion parameters and only 20,000 samples, the random model should have failed to learn or memorized everything; instead, its curves looked “almost like you’re training MNIST,” and it overfit less aggressively than an MLP on MNIST.
The scale sharpened the surprise. Computer vision had already shown implicit regularization when a 50-million-parameter ImageNet model learns from one million examples—a 50-to-1 ratio—but 7 billion parameters against 20,000 samples is vastly more extreme. Transformers are also known to overfit more readily than ResNets in vision, yet this causal language-model architecture exhibited a strong implicit bias against overfitting.
What the model actually uses remains open. Balestriero points to attention entropy as one possible mechanistic-interpretability lens and asks whether neural collapse or a lottery-ticket-like subnetwork explains the behavior. He wants to probe which parameters are useful, whether every layer contributes, and whether only the final layers learn the task.
The host’s summary—that teams might return to specialized models—earned a qualified agreement. For a small number of fixed tasks, Balestriero expects purpose-trained models with a suitable architecture and supervised data to work well, but only if deployment “will never go too out of distribution.” The strongest conclusion is existential: there are tasks where next-token prediction “is not the answer” and is “not better than random initialization.”
2. Pretraining buys breadth, generation, and behavioral flexibility
The cheap specialist does not generalize freely. A model trained to classify job descriptions may identify an occupation correctly but cannot infer which occupation pays more if that comparison never appeared in training. Pretraining’s value therefore rises with open-ended classification, reasoning, new scenarios, and the need to generalize beyond the supervised distribution.
Generation is a categorical boundary: “For generation, there is no question that this is what you need to do.” A classifier returning only “good” or “bad” also gives less flexibility for interaction and behavioral improvement than a generative model whose answers can be challenged, collected, and turned into fine-tuning data; guardrails would have to take the form of further supervised training.
Balestriero’s “FAIR Language Model Paradox” supplies the cost mechanism. Next-token models must learn rare tokens across the full distribution, making training slow and wasteful; a supervised classifier can ignore rare tokens irrelevant to its target. In practice, the team used ordinary supervised training “out of the box,” without hyperparameter optimization, and observed “very, very stable training.”
3. Multitask objectives could occupy the useful middle
The host asked about an intermediate solution between universal pretraining and one-task training, such as five representative tasks. Balestriero’s answer: these objectives form a continuum, because next-token prediction itself can be viewed as many binary tasks asking whether each candidate is the next token.
The research problem is to design “the minimum number of tasks” that produces the most diverse representation. A carefully selected multitask objective might support new tasks arriving “on the go” without paying to recover every detail of the token distribution; expected task variety would determine the appropriate point on the spectrum.
The discussion also questioned whether generation benchmarks and exact next-token prediction capture understanding. Balestriero argues that meaningful prediction need not specify every granular detail: in “I saw this movie for ___ minutes,” it may suffice to represent a time component rather than generate “52 minutes.” Because tokenization is “a lossless compression” and close to raw data, he says a concept token would be more meaningful.
4. SSL is a general relation framework, not supervised learning’s opposite
In “The Birth of Self-Supervised Learning: A Supervised Learning Perspective,” Balestriero and Yann Le Cun show how a supervised objective such as least squares can become an SSL objective comparing samples. Predicting “car or dog” and asking whether two images represent the same thing can learn equivalent representations, up to symmetries irrelevant to linear probing.
SSL generalizes better because its implicit labels are much finer-grained—not because its loss possesses a separate magic. In the limit, “each image is its own class,” preventing distinct images from collapsing together and preserving distinctions that future downstream tasks might need.
The host also asked how SSL could maximize worst-case downstream task performance, but the excerpt moves immediately to class-balance effects rather than providing a direct sketch of that argument. The transcript therefore does not support a stronger worst-case claim.
Balestriero’s hierarchy is explicit: “SSL is more generalized than supervised learning.” Labels, adjacent video frames, or other prior knowledge merely provide different ways to construct the pairwise relation matrix. The central design question is no longer which camp a method belongs to, but “how do you build this pairwise relation matrix?”
5. Unified theory exposes imbalance—and Earth models expose its stakes
The correspondence maps least-squares supervision to VCREG, with variations such as VICReg or W-MSE, and cross-entropy toward SimCLR-like losses. That lets SSL reuse a large body of supervised-learning theory: neural-collapse results transfer in “five lines,” while semi-supervised objectives can be weighted according to the relative sample counts instead of simply adding a coefficient and choosing it through cross-validation.
Current SSL objectives implicitly assume balanced concepts. That works relatively well on ImageNet but creates a huge representation bias on heavy-tailed datasets such as iNaturalist. Rather than discarding oversampled concepts through curation, the supervised correspondence yields a principled SSL reweighting scheme and can potentially incorporate varying sample noise.
The same average-versus-tail problem appears in implicit neural representations of Earth data. A model given a location and date can interpolate temperature or precipitation where sensors are missing, yet predictions around islands and coastlines may be “almost random.” Architecture explains a substantial share of the disparity, though Balestriero hedges that irreducible uncertainty and sparse observations may prevent bias from disappearing entirely.
Fourier bases improve on using no basis, but impose stationarity and weak localization—poor assumptions where temperature or precipitation changes sharply. Wavelets localize better and remove some bias, although “wavelets are not the answer to everything”; the longer-term goal is to let models learn the appropriate basis from data rather than fixing it in advance.
Crowdsourcing compounds geographic bias because observations scale with local user populations. Balestriero floats a conditional mix—perhaps 10% high-quality, uniformly sampled data anchoring 90% crowdsourced data—while stressing that the right distribution depends on the application. A housing model may legitimately prioritize populated areas, whereas environmental or ecosystem-oriented uses may require coverage beyond human-populated areas.
Accountability, in his framing, belongs in a downstream feedback loop. Once the intended policy use is known, downstream users should establish location-specific evaluations, report failure modes, and iterate with model designers until the system is trustworthy. Global state-of-the-art scores can otherwise hide the same geographic blind spots seen in vision models trained predominantly on North American chairs and cars.