The Data Factory: Inside the $100B Race for Post-Training Supremacy, with Labelbox CEO Manu Sharma
Summary
The Meta–Scale AI deal highlights data as a third strategic input alongside compute and research talent. Manu Sharma says each frontier lab is “probably spending over a billion dollars a year on data,” with an “insatiable appetite” for specialized datasets. Labelbox operates a vertically integrated data factory that can rapidly deliver expert-built reinforcement-learning assets.
Post-training is shifting from imitating human reasoning traces to letting models practice inside expert-designed “gyms.” Humans increasingly formulate representative problems, solutions, rubrics, verifiers, and autograders; models then learn through repeated attempts and reward rather than copying a person’s narrated thought process. Sharma’s core formulation is “modeling the problem and a solution and how would you grade the solution.”
Coding and mathematics are advancing fastest because their rewards can often be verified mechanically. A coding agent can attempt a pull request millions of times and be scored against old and new tests, while many mathematical answers can be checked numerically; Sharma attributes much of the prior nine months’ reasoning and coding progress to this dynamic. The next commercial frontier is transferring RL to domains where several answers might be valid and “what is good and great or excellent” must be expressed by experts.
Frontier models look convergent from a distance but diverge materially once an expert pushes them to their limits. Sharma reaches for Gemini on algorithmic, math-heavy coding and Claude for refactoring or distributed-system architecture, where it gathers context and asks more questions; he can feel when he has “hit the ceiling of this model.” Labs may aspire to universal systems, but current products are being optimized around different users, workflows, and quality targets.
Enterprise fine-tuning is narrowing to cost optimization and genuinely proprietary judgment, while context engineering becomes the default implementation layer. Sharma sees large-model outputs being distilled into cheaper small models, or weights changed where no base model captures the company’s desired judgment; otherwise, retrieval, prompts, tools, and task-specific context usually win. Fine-tuning can mean “freezing an investment” while foundation models improve every few weeks.
The new expert-data economy bears little resemblance to low-paid, five-minute labeling work. Labelbox says its top contributors earn “well north of $250,000 a year,” while its network spans more than 70 countries and the company conducts well over 2,000 AI interviews daily. Some increasingly complex tasks involve building RL environments, solvers, verifiers, and representative trajectories, with synthetic generation accelerating—but not replacing—human judgment.
Fast AI takeoff and persistent human supervision may both be true because capability scales faster than models acquire taste. Sharma believes “we are already in an accelerated takeoff,” with roughly 30 million developers today and potentially billions of coders using AI assistants, yet sees effectively “infinite slices” of human knowledge still to encode. The unresolved bottleneck is quality: humans often cannot explain why something is excellent, but “we know when we see it.”
Deep dive
1. Meta’s Scale AI move exposes data as frontier infrastructure
Labenz opens from the industry shock: Meta’s recently announced deal with Scale AI, Alex Wang’s move to lead its superintelligence effort, departures from Scale, and teams reorienting their supply chains. Sharma says M&A can shift workloads, sometimes quickly, and expects more industry changes.
Sharma’s strategic stack has three components: compute, visible in extraordinary capital expenditure; researchers who understand architectures, training recipes, and what to train on; and data, which he describes as equally important. His description of the present moment—“the maximum innovations we’ve ever seen in per day or per week”—rests on an AGI race involving both major labs and smaller teams betting on alternative techniques.
Sharma says Labelbox has been behind the scenes providing datasets to most of these AI labs. He expects new teams and methods to keep changing how specialized datasets are created and delivered.
2. Post-training has become the layer that turns knowledge into products
Sharma traces three regimes. Labelbox began in 2018 amid supervised learning, when people tagged images, video, and text for models to imitate; transformers shifted learning toward vast unsupervised datasets; then SFT and RLHF supplied the specialized behavior that made strong base models useful to knowledge workers.
Pre-training still takes months—Sharma uses six months as an illustrative release-cycle constraint—and labs continue producing gigantic base models. Those can operate as “teacher models” for smaller “student models” that actually ship, but a growing share of budgets is moving into post-training because raw knowledge does not itself produce reliable assistants or agents.
Labenz offers the Claude 4 anecdote: an omitted harmful-system-prompt dataset created the wrong behavior, and Anthropic apparently patched the issue later rather than rerunning post-training. Sharma does not provide a compute percentage, but says leading labs and hyperscalers are researching the full end-to-end training recipe and that post-training budgets are growing.
The core capability gap is long-horizon work. A base model may understand software, but building an entire product suite with minimal supervision requires deliberately teaching the model how to perform such tasks reliably—capabilities Sharma says are beginning to appear but “it’s not there yet.”
3. Verifiable rewards explain the coding-and-math surge
Coding and mathematics are the obvious RL beachheads because success can often be scored. For a pull request, the system can check whether the stated objective was met and whether both existing and new tests pass; for many math problems, a numerical result supplies an unambiguous reward.
Inside training, a model may make “millions of attempts” at a task. Successful attempts score well, weights change, and the system advances—an automated practice loop that Sharma links to the striking gains in reasoning and coding during roughly the previous nine months.
Coding is doubly valuable: it is lucrative knowledge work, and some labs believe stronger coding systems will accelerate their own AI research. That could create a feedback loop between commercially useful agents and faster model development, even before systems can autonomously perform large-scale professional software engineering.
4. Apparent model convergence hides expert-level specialization
Sharma grants Labenz’s surface observation: leading systems increasingly share similar chat interfaces and generic capabilities. His qualification is that “to be able to really critique a capability, you really have to be at an expert level”; meaningful divergence emerges only when someone can judge the nuances of difficult work.
In his own coding, Sharma may prefer Gemini for an algorithmic problem requiring mathematics. For refactoring or distributed-system architecture, he may choose Claude because it reads multiple files, gathers missing context, asks questions internally, and eventually makes a better architectural decision.
The practical test is whether an expert senses a stalled trajectory: “I’ve hit the ceiling of this model.” That judgment reveals differences that casual use can obscure.
Labenz adds that product strategy reinforces divergence. He sees Anthropic intentionally prioritizing coding, while other labs may optimize for consumer tasks or a general social assistant; whether the best coding, research, and everyday systems can later be recombined into “a single world model” remains unresolved.
5. Human data anchors both competence and intended behavior
Labenz contrasts DeepSeek-R1-style reinforcement learning with human-grounded training: verifiable reasoning can emerge without human traces, but may bring strange behavior such as language switching. He also contrasts GPT-4.5’s strong trivia knowledge with o3-mini’s superior reasoning, showing that world knowledge, reasoning, and behavior can be separated.
Sharma argues that human intelligence is routinely underestimated. Models are superhuman on selected slices yet unable to perform many ordinary tasks reliably; training therefore remains an attempt to emulate pieces of the vast economy, institutions, and workflows humans have already built.
His Wi-Fi support example captures the gap. A useful agent must converse naturally, identify the customer’s problem, route to specialized expertise, navigate company software and databases, perhaps provision a replacement, and handle tens of millions of users’ edge cases—not merely answer a question correctly.
Synthetic methods can bootstrap and accelerate those datasets, but “you can’t really invent these assets from algorithmic synthetic approaches.” Personality, service standards, acceptable trade-offs, and the desired customer experience are design parameters chosen by the organization, so they require human quality judgment.
6. Reasoning traces are yielding to problems, rubrics, and gyms
Sharma says researchers spent considerable effort asking academics and other experts to narrate step-by-step reasoning. The result was difficult to collect faithfully: people are often poor at expressing how they actually think, despite earlier work on process-reward models designed to supervise intermediate steps.
His description of the new center of gravity is “modeling the problem and a solution and how would you grade the solution.” Reasoning then becomes an emergent property of RL, though Sharma preserves the hedge that traces may still need to be generated or audited.
Labenz tests the formulation: provide a solution and rubric, let the model attempt the task, and have another model assign reward against those criteria. Sharma agrees but adds the essential non-verifiable case—experts often do not know the one right answer, yet can recognize “what is good and great or excellent.”
RL environments are best pictured as “a gym”: a model repeatedly practices until it acquires a skill. Games may teach generalized planning, while coding environments can be domain-specific; the difficult work is building representative tasks and rock-solid autograders whose scoring aligns with expert judgment.
7. Long context must be tested as work, not search
Programmatic needle-in-a-haystack tests establish whether a model can retrieve a hash or fact from a large volume of noise. Sharma treats that as a useful but fundamental-level measurement, not evidence that the model understands a very long professional context.
His stronger specimen is a financial analyst synthesizing SEC filings, company disclosures, and earnings-call audio. The task is not to locate one figure across ten documents; it is to build a company model and forecast earnings potential, a real workflow that may occupy a skilled human for days.
Labelbox calls these “multi-hop” capabilities: reasoning across many pieces of information and modalities. Sharma says Gemini is notably good at locating moments in video, but frontier models can still be induced to fail on representative industry tasks; those failure cases become especially valuable hill-climbing data.
8. Enterprise AI succeeds through iterative evals before training scale
Labenz recalls fine-tuning the text-davinci-002 series in 2022: create ten examples, train, have the model attempt the next hundred, reject or correct failures, feed the difficult cases back, and repeat for “two to N rounds” until performance became acceptable.
Sharma calls that iterative evaluation pattern best practice. Teams need a holdout representation of the workflow, then must expand it as they discover “known knowns” and “unknown unknowns”; the recurring failure mode is not insufficient model sophistication but failing to define what the intended automation actually does.
Producing evaluation datasets is becoming a craft analogous to software development. Edge cases force teams back to the drawing board, and successful organizations treat data and evals as iterative products rather than one-off inputs.
This also resists universal dataset-size rules. Narrow tasks may need surprisingly little data, but broader input distributions require more; the decisive question is whether the examples represent the real workflow and its failures, not whether a nominal example count has been reached.
9. Fine-tuning is retreating to efficiency and proprietary judgment
Sharma’s 2025 view is blunt: “fine-tuning at best is helping you make the model efficient” for many tasks. A large frontier model may achieve the required quality but cost too much at millions of queries per hour, so its outputs can be distilled into a smaller model delivering similar quality at a fraction of the cost.
A second category survives where the base model truly lacks a capability or the company holds a distinctive quality judgment. Labelbox fine-tunes its own assessment systems because an off-the-shelf model’s opinion of whether an interviewee will produce excellent frontier-training data does not necessarily match Labelbox’s observed outcomes.
Sharma cites what he believes was Google’s Med-PaLM as a warning against overvaluing narrow healthcare corpora: based on something he read, he says the base Gemini 2.5 models outperform the hyper-specialized healthcare model. His inference is that reasoning learned across broad data can matter more than petabytes of proprietary information that do not improve reasoning itself.
In open source, he most often sees Qwen, Llama, and Mistral selected for fine-tuning. The choice often follows operational constraints: specialized companies may need air-gapped or self-hosted deployment, whereas digital-native companies can use state-of-the-art models from cloud providers or OpenAI.
10. Context engineering is becoming the enterprise control plane
Context engineering extends beyond prompt wording to retrieval, relevant examples, files, tools, and the entire information package supplied at runtime. Cursor is Sharma’s canonical example: coding works because the base model receives directory structure, relevant snippets, related functions, and repository context for the task at hand.
The architecture generalizes: define the task, retrieve the right information, and expose the functions required to act. Sharma sees enterprise agents as software-engineering systems with domain-task mapping, often involving several tool calls and trajectories—but without requiring every company to become expert in neural-network training.
Labenz stress-tests the advice on his podcast introductions. He supplies Claude with the current transcript and previous finished essays, but not the source transcript behind every example; Claude reproduces cadence and his mix of AI enthusiasm and fear, yet often misses what is uniquely timely or intellectually central.
Sharma recommends persistent rules, examples of failures and gotchas, alternative base models, background on the people involved, and tools—not freezing the workflow into a fine-tune while capabilities change every few weeks. His deeper advice is editorial: perhaps the optimal system keeps Labenz as “editor-in-chief,” using AI for leverage without automating the distinctive creative judgment.
11. Frontier demand is creating a vertically integrated expert economy
Sharma describes frontier-lab demand as an “insatiable appetite” for novel datasets that teach longer-horizon tasks across far more than coding and mathematics. Each lab is “probably spending over a billion dollars a year on data,” and he says those budgets are increasing.
Staffing agencies can rapidly place contractors, but a data factory must also vet contributors, build tooling, orchestrate workflows, assure quality, and deliver finished datasets quickly. Most current datasets are hybrid: AI and synthetic-generation techniques scale production, while human experts ground the resulting data.
Labelbox’s top contributors earn “well north of $250,000 a year,” versus an approximate $40,000-$50,000 asymptote across other domains and countries. The tasks have evolved from five-minute annotations to building RL environments, solvers, verifiers, or examples of good activity; Sharma expects compensation to rise as training enters business workflows.
Enterprises use a different operating model. A pharmaceutical, medical-imaging, or medical-coding company often needs its own deeply specialized people and runs the factory using Labelbox’s software platform; Palantir-style forward-deployed assistance depends on customer sophistication, with strong AI startups more likely to retain full architectural control.
12. Sovereign AI should begin with citizen outcomes, not data hoarding
Labenz asks whether Brazil, India, or Mexico could collect culturally and linguistically relevant data, hand it to frontier developers, and improve local performance without funding data centers, researchers, and national foundation models. Sharma resists a data-first prescription: governments should define the desired service, then work backward.
The opportunity is an “AI-first” redesign of public services at lower cost—citizens discovering available benefits conversationally, accessing basic healthcare services, or filing taxes through a more intuitive interface. Depending on the country, government may own the stack or specify outcomes for private providers.
Sharma’s Planet Labs experience supplies the concrete analogy: Planet operated more than 400 low-Earth-orbit satellites that scanned the Earth daily, allowing Brazil to measure deforestation without operating its own constellation. Some capabilities should be purchased from the specialist best positioned to produce them; others may be uniquely curated by governments for their citizens.
Labenz presses for the shortcut, but Sharma says compute, data, and talent ultimately remain necessary in many cases. Projects fail when institutions acquire technologies and build abstract platforms before defining the experience; once the outcome is explicit, producing datasets, evaluating systems, and deciding between applications or specialized models becomes tractable.
13. Computer-use agents will turn real workflows into training supply
Asked why nobody has paid to watch him use his computer, Labenz gets a direct answer: “It’s coming.” Sharma says Labelbox already powers data creation for multiple computer-use agents, while current products remain in the “first innings” of the companion imagined in Her—listening, seeing, and understanding ongoing work.
Present models may read screens well yet fail on geometry, spatial relationships, or time. Sharma’s telling failure occurs three minutes into a session, when the model no longer preserves the goal established during minute one; competence on a screenshot is not competence across a long workflow.
Sharma would not be surprised if some premium products used customers’ sessions to improve reliability, subject to product and privacy choices. Meanwhile, Labelbox is producing sessions across languages and specialties at scale: “the capabilities are being cooked. It’s not rolled out yet.”
14. AI interviews industrialize screening—and expose new games
Labelbox looks beyond formal credentials for “high agency and IQ,” which Sharma believes predicts performance on unfamiliar, generalized, long-horizon tasks. A capable generalist can still generate training signal precisely because they can learn work that current models cannot complete reliably.
Its Zara interview system conducts well north of 2,000 sessions daily, with probably 50 people interviewing at a given time. Thirty-minute conversations incorporate résumés, publications, research, domain expertise, and language fluency; contributors rate the experience around 4.6-4.7 because it is patient, available on demand, and can discuss technical work beyond a general recruiter’s depth.
Demand for practice interviews emerged from contributors rather than Labelbox’s original plan. The less flattering emergent behavior is gaming: Sharma has seen people place an iPhone running ChatGPT Advanced Voice nearby so two AIs converse, forcing the company to improve assessment and identify participants with the right intent.
Multiple model providers power the conversation, while Labelbox uses fine-tuned capabilities for downstream grading. Interview outcomes feed back into assessments of who actually produces strong data, creating a closed loop that changes both scoring and the interview format over time.
15. Capability may take off quickly while quality remains human-grounded
Sharma sees effectively “infinite slices” of human intelligence left to emulate. Across supervised learning, unsupervised pre-training, RLHF, and the new RL paradigm, techniques have changed repeatedly while data has remained essential; even synthetic datasets require human grounding if systems are ultimately meant to serve people.
His long-term picture keeps humans “in the driver’s seat,” commanding or managing millions of AIs through data. The data may take forms not yet invented, but it remains the medium through which people specify tasks, acceptable behavior, and quality.
Labenz frames the closing tension: a best-human coder replicated in massive parallel could accelerate architectural discovery and trigger fast superintelligence, while millions of domain-specific slices and datasets could produce a plateau near expert-human performance. Sharma answers that both are true—“we are already in an accelerated takeoff”—even as daily familiarity makes the progress feel ordinary.
Sharma starts from roughly 30 million developers today and imagines billions of coders using AI assistants over the next few years, but identifies quality judgment as the unresolved constraint. Referencing Zen and the Art of Motorcycle Maintenance, he notes that people often cannot articulate why something is good or bad: “we know when we see it.” Whether AI can acquire that taste may determine the next acceleration.