Training the AIs' Eyes: How Roboflow is Making the Real World Programmable, with CEO Joseph Nelson
Summary
Computer vision is approaching its “ChatGPT moment,” but remains roughly three years behind language because the physical world has a much fatter tail than human-authored text. The vision transformer arrived in 2020, three years after the original transformer, and models are steadily pushing from familiar people, cars, and food into rarer scenes. Joseph Nelson expects a “Cambrian explosion” as visual understanding becomes a baseline product expectation.
Frontier multimodal models remain unreliable precisely where industrial vision demands precision: grounding, spatial relationships, measurement, speed, and reproducibility. On Roboflow’s RF100-VL benchmark, the best tested model, Gemini 2, achieved only 12.5% across real user domains; adding one to five examples in RF20 lifted the best model by around 10% at most. Gemini 3 could label Nelson’s data but sometimes took 40 seconds per image, while identical prompts could produce non-reproducible outputs.
The production opportunity lies in distilling broad frontier capabilities into owned, task-specific models that meet edge economics. For Wimbledon instant replay, SAM 3 can label historical footage for players, balls, courts, and nets, but deploying that general model is uneconomic under what Nelson describes as a “sub-10 nanoseconds” transmission constraint. A smaller RF-DETR model can run multiple video streams on one A100 because it does not need to recognize poker chips, deer, or anything else irrelevant to tennis.
Accuracy requirements are business requirements, so data and compute needs vary by orders of magnitude. An 80%-accurate museum-traffic counter can be valuable when the alternative is no visibility; particulate detection in an IV bag demands extremely high recall because lives are involved. Controlled manufacturing tasks may work with hundreds of images, while Nelson says even petabytes are insufficient to capture autonomous driving’s stop-sign long tail.
Roboflow’s technical wedge is shared-weight neural architecture search, which turns a limited compute budget into a family of “one-of-one” models along a speed-accuracy frontier. One training run samples thousands of subnetworks across patch size, decoder count, queries, resolution, and attention configuration instead of separately training each configuration. Built from a Meta DINOv2 backbone, Apache 2.0-licensed RF-DETR spans nano through 2XL; Nelson says 2XL is more accurate than a fine-tuned SAM 3 and 40× faster.
Open-source vision is strategically exposed to Meta, while Chinese labs have consistently led much of the field. Nelson highlights Alibaba’s Qwen3-VL, GLM’s 9-billion-parameter work, and DeepSeek’s OCR research; in the US, Meta’s lineage from Detectron2 through DINO and SAM 3 remains foundational. Roboflow has raised about $63 million across all rounds—meaningful but far below foundation-model budgets—and Nelson sees NVIDIA’s expanding open-source investment as the likeliest counterweight if Meta changes course.
Coding agents could expand Roboflow’s market by converting difficult model-building workflows into intent-driven infrastructure consumption. Skills, CLIs, and potentially MCP let Claude Code, Codex, and other agents curate data, train models, and choose a Pareto point; an unreleased first-party agent is intended to act like an expert “guide and sherpa.” The harder task may be clarifying whether “count cars crossing the line” means the correct car class, line, and intersection—not invoking the model once intent is known.
The next S-curves are world models, vision-language-action models, inference-time visual agents, and wearables—but regulation could determine how broadly their value compounds. Nelson says 8 million pairs of smart glasses sold last year versus 60 million AirPods, evidence that hardware adoption has begun even though onboard AI “is not there.” He rejects regulating model size or imposing narrow-model mandates, arguing for liability at the outcome level; Labenz separately frames the risk of getting “all the weapons and don’t get the energy.”
Deep dive
1. Vision is nearing its product breakout after a three-year architectural lag
Nelson dates language’s lead to “Attention Is All You Need” in 2017, followed by five years of scaling laws, GPT-2, GPT-3, GPT-4, and ChatGPT in 2022. The vision transformer arrived in 2020, creating an analogous gap before broadly useful visual products.
His biological analogy explains why one giant reasoning model may not dominate every visual workload: humans pair a large reasoning engine with rods, cones, and a visual cortex optimized for “lizard brain,” fast-reaction perception. Machine systems may similarly combine general reasoning with specialized, low-latency visual components.
Language and vision are “crashing together” because each supplies context the other lacks, but language is a human construct optimized for human interpretation. The physical world is not: “The world’s much bigger than just language,” with more distinct daily scenes than unique words a person is likely to read.
Roboflow’s demand sample spans about a million developers downloading open source every 30 days and roughly half the Fortune 100 building on the platform. Production ranges from Wimbledon instant replay and Rivian quality assurance to a Canadian YouTuber’s flamethrower, weed-killing robot—early evidence for Nelson’s expected “Cambrian explosion.”
2. The center of the visual distribution is solved; the fat tail is not
Labenz proposes a demanding definition of “solved”: could nearly any vision problem be completed today if one accepted frontier-cloud costs, latency, and substantial engineering? Nelson’s answer is no because a solved problem should require little more than asking: “It almost impresses me. It delights me that it already understands.”
Tasks near the distribution’s center increasingly qualify. People counting is mature, while OCR is approaching the point where a real-time model can answer “How much was my salsa from this receipt?” or identify the house number on the left side of a Street View image.
Nelson’s first-principles explanation starts with representation: Unicode stores text far more compactly than a single RGB image containing three channels, values from 0 to 255, and pixel-by-pixel structure. That greater informational heterogeneity produces “fatter” long tails of unusual viewpoints, objects, relationships, and environments.
Models are pushing outward along this bell curve, but edge deployment trails a newly demonstrated cloud capability by perhaps 18 months. A task may therefore be technically tractable yet commercially unsolved when it cannot run on an iPhone or Jetson Orin-class device at the required latency.
3. Grounding and precision remain frontier models’ most revealing weaknesses
Roboflow maintains VisionCheckup.com to highlight multimodal-model failures in spatial reasoning, precision measurement, and grounding. Performance keeps improving, but failures persist in grounding: locating, detecting, or segmenting the exact pixels associated with an instruction rather than merely describing the scene plausibly.
Labenz recalls ARC-AGI puzzles where models could not reliably describe the initial colored-grid state, making later reasoning failures unsurprising. Nelson sees the same issue in crosswords: a model can solve a clue from known letters as text, yet struggle to identify where the answer belongs in the visual grid.
Measurement and spatial relations are especially brittle because they demand precision rather than a broadly helpful response. Nelson believes labs may prioritize inferring user intent and calling specialized tools over solving pixel-perfect segmentation directly, but the segmentation component of that reasoning chain remains “pretty unsolved.”
More reasoning and expert tool calls can improve results, yet they introduce two other problems. Nelson’s Gemini 3 labeling attempt took about 40 seconds per image, and failures involving SAM 3, Gemini, and Claude Opus sometimes could not be reproduced because the generative models did not reproduce the same way.
4. Real-world benchmarks show few-shot prompting helps without closing the gap
RF100-VL draws 100 datasets from the hundreds of thousands of open projects shared through Roboflow, covering industrial, health-care, flora-and-fauna, document, and miscellaneous tasks. The test asks models to perform segmentation from instructions comparable to those given to a human annotator.
At publication, Gemini 2 led the evaluated multimodal and specialist models with just 12.5% performance across all domains. Nelson stresses that these were not arbitrary academic examples: “These are the places that folks are using models.”
For a CVPR competition, Roboflow reduced the basket to RF20 for compute reasons and supplied one, two, three, four, or five image examples. The maximum lift for one model was around 10%; Nelson hedges that he would need to check the average across every domain.
That gain is meaningful when the zero-shot baseline begins near 12%, but it is “not a panacea.” Nelson remains bullish on image-text pairs and visual examples as prompts; they ground the model in a domain without replacing the need for better representations of real production scenes.
5. Deployment starts with requirements, not a universal model leaderboard
Nelson organizes model selection around the “job to be done.” Accuracy and speed form the familiar Pareto curve, but ownership, privacy, security, intellectual-property creation, local operation, and the cost of continuously transmitting video add dimensions that cannot be reduced to a single benchmark.
Real-time inspection or live broadcasting usually forces compute close to the camera. Once a model must run at the edge, the operator generally needs the weights and deployment control—one reason Roboflow invests heavily in open models and sees open source as essential to bottom-up use-case discovery.
His recommended sequence is pragmatic: test whether a frontier model makes the problem tractable, “use a model to failure,” and then decide what must be owned, distilled, fine-tuned, or surrounded by pre- and post-processing. The ceiling model tests possibility; production requirements determine the eventual architecture.
6. Wimbledon shows how general intelligence becomes an efficient specialist
The instant-replay provider Nelson describes takes compute directly to Wimbledon and US Open courts because the broadcast is live and, in his words, has “sub-10 nanoseconds to put something over the wire.” Even where hardware could run SAM 3, its compute load was not economic for this job.
Historical tournament video can instead be prompted through SAM 3 to label people, tennis balls, courts, and nets. That creates a curated dataset for a fixed visual vocabulary; poker chips and deer are possible SAM 3 prompts, but Nelson says their odds of appearing at Wimbledon, and their relevance to the replay model, are very limited.
The resulting data trains RF-DETR, a smaller model designed for real-time detection and segmentation. Nelson says it is efficient enough to process multiple streams on one A100, converting frontier-model capability into lower latency, lower cost, and a model the operator owns.
He rejects any stigma around conventional code in the harness. Rather than asking Gemini to return a player count, a detector can return two person instances and “a tiny bit of logic” can count them—often a faster and more reproducible system than asking one generative model to absorb every operation.
7. Data requirements scale with scene variation and the cost of errors
A museum may accept an 80%-accurate people counter when its alternative is no visibility at all; distinguishing roughly a dozen visitors from 100 can already improve staffing. “Ship it” depends on incremental business utility, not an abstract demand for perfect perception.
An IV-bag manufacturer faces the opposite loss function. Escaped particulate matter may threaten a patient’s life and the company’s reputation, so the vision system needs extremely high recall and will initially augment human inspection, lab inspections, or a sampling methodology rather than casually replace them.
Nelson invokes Andrej Karpathy’s stop-sign examples to show why driving consumes enormous data: a sign can be hidden by a bush, mounted on a school bus, attached to a raised gate, or qualified by turning instructions. “Petabytes isn’t sufficient for a car,” while a controlled battery-production scan may yield value from hundreds of images.
Model sizing follows the same intuition. RF-DETR ranges from nano, small, medium, large, XL, to 2XL; Nelson says fine-tuned 2XL beats fine-tuned SAM 3 while running 40× faster, while pico or nano variants can exceed 180 frames per second on a 4 GB Jetson Nano.
8. China leads open vision while the US ecosystem leans heavily on Meta
Nelson argues that the US has “almost never led” visual AI as consistently as it has language. China’s manufacturing-heavy economy creates stronger incentives to prioritize machine vision, and Alibaba’s Qwen team has produced Qwen3-VL capabilities he considers competitive even with closed visual-language models.
He also highlights GLM’s 9-billion-parameter mixture-of-experts work, especially for OCR, and DeepSeek’s experiment using page screenshots to compress more text into training. The loss of understanding was substantially smaller than the token compression gained, turning vision into a data-processing technique for language-model scaling.
The US bench includes Meta’s SAM 3, Microsoft’s Phi family, the Allen Institute’s multimodal work, and NVIDIA’s growing investments, including Cosmos Reason. Nelson particularly credits Meta’s lineage from Detectron2 and Faster R-CNN through DETR, DINO, and Segment Anything—work overshadowed by criticism of its language models.
The fragility is cumulative: open research progresses by combining ideas, running ablations, and testing shared components. Nelson says a change in Meta or NVIDIA’s open-source posture would cause “all of open source” to take a hit. He notes recent changes among Qwen’s leading researchers, while Labenz raised the possibility of a Meta strategy change; NVIDIA’s repository volume and multimodal direction give Nelson optimism.
9. RF-DETR proves a focused lab can compete without hyperscaler capital
Roboflow chose a narrow, under-addressed task: real-time object detection and instance segmentation for owned edge deployments. Nelson calls RF-DETR the first real-time instance-segmentation transformer and says it retook the state of the art for Roboflow in this specific speed-accuracy category.
The model begins with Meta’s DINOv2 pretrained backbone, then searches for configurations that convert transformer accuracy advances into real-time speed. It uses an Objects365 pretrain, transfers downstream to COCO and customer fine-tunes, and attaches both detection and segmentation heads.
Roboflow began the experiments after its Series B and has raised about $63 million across all rounds. Nelson calls that “not nothing,” but it covers the whole company and “pales in comparison” with the billions available to frontier-model builders—making narrow task selection and compute efficiency essential.
Detection arrived in April and segmentation in the fall. RF-DETR is Apache 2.0-licensed, allowing commercial building where some supported alternatives require commercial licensing; Nelson says China’s LW-DETR team has responded without yet beating the work, creating a “cool kind of global arms race.”
10. Shared-weight search turns one training run into thousands of models
Conventional architecture search repeatedly trains models across an informed grid, measuring speed and accuracy after each expensive guess. Roboflow’s weight-sharing approach instead trains thousands of subnetwork configurations in parallel inside one run, making the search feasible within its compute budget.
At every training step, the system samples a subnet by varying patch size, decoder count, query count, input resolution, attention windowing, and other parameters. At inference, any sampled subnet can be evaluated, yielding a family of models rather than a single architecture.
A NAS run is less efficient than one ordinary training run, but it is not thousands of times more inefficient despite comparing thousands of configurations. Nelson calls this the “huge freaking unlock”: Roboflow gained a reusable framework for releasing model families, not merely one successful checkpoint.
Deformable attention required inference-engine work and eventually TensorRT support; DINOv3 naturally suggests the next experiments. Roboflow has also productized hosted NAS for customer datasets, returning a Pareto curve from which users select their speed-accuracy point—an architecture so tailored that Nelson calls it a literal “one-of-one.”
11. Agents turn model infrastructure into a guided outcome
For human users, the NAS “easy button” hides subnet orchestration: launch training, receive the curve, and click the desired operating point. Nelson’s product principle is good defaults with progressively revealed complexity, while keeping the underlying paper, models, and benchmarks open enough to reproduce.
Roboflow applies the same philosophy to inference. Its open repository can be installed directly, while the managed product encodes optimizations such as running image resizing on CPU and reserving GPU capacity for the network components that require it.
Claude Code, Codex, and other agents can use Roboflow’s CLIs to act on platform workflows, including optimizing models; Nelson remains open on the “MCP or CLI future,” saying robust CLIs already let agents act on a user’s behalf while a workflow-specific MCP may follow.
An unreleased first-party agent would sit above the infrastructure, inspired by companies that layered generative interfaces over strong cloud primitives. Given “count cars crossing the line,” it could choose an efficient pretrained model, then ask which line or intersection the user meant—the difficult part being intent discovery, not the final invocation.
12. Aesthetic judgment resists the benchmarks that drive scaling
Labenz’s Waymark problem separates obvious extremes from the difficult middle: models can reject terrible small-business advertising images, but ranking several plausible options by “what would make the business proud” remains slow, expensive, and unstable. Frontier models help, yet no small open model clearly solves the taste problem.
Nelson’s mechanism is blunt: recursive improvement works when a capability can be benchmarked, after which compute and the “bitter lesson” can take hold. Aesthetics remains “eye of the beholder”; people prefer the recognizable looks of Midjourney, ChatGPT, or Gemini without agreeing on one objective target.
Rules are easier than taste. A system can measure whether text occupies too much of a display ad, while “does this ad feel good?” lacks comparable ground truth. Nelson points to the LAION aesthetics predictor as relevant prior work but offers “no great zero-shot aesthetics model” beyond Labenz’s existing foundation-model approach.
His best path is client-specific preference learning: brand guidelines, past ads, and client preferences could produce an RLHF-style “vibe check” or few-shot evaluator. Yet the method contains its own contradiction—matching historical brand taste may fail marketing’s higher-order objective of being different enough to “stand out from the noise.”
13. Self-supervision, world models, and VLAs define the next capability curves
The established curve is “transformer everything”: vision transformers and diffusion transformers continue consuming tasks and setting state-of-the-art results, with RF-DETR extending the recipe into real time. The newer curve is self-supervision, where DINO-family backbones develop rich scene representations without conventional labeled datasets.
Nelson recalls DINOv3 using billion-scale or perhaps billion-plus images but explicitly says he would need to verify the number. A teacher validates a student on tasks such as patch prediction or fill-in-the-blank, producing latent understanding that can later support classification, segmentation, detection, or captioning; treating patches independently also helps explain persistent spatial-reasoning weaknesses.
World models could blend “read-write access” to physical scenes: understanding environments, predicting what comes next, and eventually supporting action. Nelson thinks they probably yield better physics and spatial reasoning, but the real uncertainty is timing; Cosmos Reason and even Sora 2 may represent early underpinnings before truly open-ended world understanding.
Vision-language-action models add explicit instructions and physical execution—“move my computer 50 cm to the left”—and will likely vary in model size and generalizability, including edge-ready variants. In parallel, inference-time scaling will create visual agents that search, call specialist models, organize images, and reason over scenes as coding agents now reason over repositories.
14. Wearables have crossed the hardware threshold before solving onboard AI
Nelson admits he has repeatedly mistimed augmented reality: Roboflow started by building AR apps before it was a company in 2017, tried again in 2019, and was wrong about the timing of adoption. The missing ingredient was not merely perception quality but a form factor that did not require a “glass brick” in the hand.
After receiving Oakley Meta glasses, he found them useful for cycling and running: they play music through bone conduction, capture media, and understand the scene. His companions did not initially know that the glasses could play music and capture media. The AI “is not there,” and the glasses appear to offload substantial work to a phone.
Still, 8 million pairs sold last year compared with 60 million AirPods—a meaningful base for a category just starting its S-curve. Nelson wants turn-by-turn directions in a future display and invokes the familiar line, “This is the worst it’ll ever be.”
He expects hardware adoption to precede software ecosystems and hopes APIs ultimately become app-store-like or allow sideloading. Snap’s Spectacles already let developers build custom lenses for scenes they want to understand—one user counted neighborhood stop signs—and Snap became the first public company to mention Roboflow in an earnings statement.
15. Nelson’s good-life vision runs from food production to bedtime
His optimistic day begins before breakfast: precision agriculture sprays only weeds, reducing pesticide use; vision checks eggs throughout the supply chain; washers stop asking humans to choose obvious settings like whites or colors; and a fridge notices missing eggs before calling the Instacart MCP.
A self-driving commute becomes faster and safer if vehicles communicate rather than react unpredictably, while connectivity lets work begin en route and returns time to family. At the office, improved visual representation could make remote collaborators feel “orders of magnitude” closer than today’s Zoom windows, though Nelson preserves a role for in-person human connection.
Evening vision systems deliver real-time football statistics and run fantasy-sports agents, while factory inspection, inventory tracking, robotics, and delivery cameras get packages produced correctly and delivered the same day. Even a toothbrush camera might scan for cavities before bed.
Nelson insists this is not theoretical: Roboflow users are already pursuing components of the chain, from cancer research and ocean cleanup to pesticide reduction and electric-vehicle quality. His playful compression is that vision will eventually “power Santa Claus”—coordinating production, verification, movement, and last-mile arrival.
16. Outcome-based regulation is the sharpest disagreement of the episode
Always-on cameras recreate an old social negotiation: smartphones initially made people uncomfortable because anyone could record them without warning. Nelson expects adoption when quality-of-life benefits outweigh new behavioral costs, but says “a jury of our peers,” not technology builders, will determine which trade-offs become socially acceptable.
His optimism rests on humanity being “inherently good even if there are bad actors,” coupled with enforceable privacy rights. Existing principles governing search, seizure, public spaces, and private spaces should be adapted to new devices rather than abandoned merely because cameras and models become more capable.
Labenz’s pushback—worth keeping—is “narrow safety through narrowness”: a public-space violence detector might sound an alarm without identifying everyone or answering arbitrary questions about them. Cost may naturally favor small models, but he wonders whether people need a new right to be processed by the narrowest model sufficient for the stated purpose.
Nelson rejects model size as the regulatory boundary because capability advances and distillation would create accidental prohibitions. His counterexample is a UNC Chapel Hill postdoc using vision to count hundreds of neutrophil colonies: broad “AI touching patient health” approval could stop benign research. He argues that fraud and other harmful outcomes should be regulated and that people should be held accountable for practicing medicine correctly, lest well-intended tool-level rules stymie useful innovation.