AI Training Data and Evaluation Benchmarks
Focuses on high-quality training datasets, cheat-proof evaluation benchmarks, and robust reinforcement learning environments.
4 sources · ↑1.1×Voices in this topic
All paragraphs
Oct 27, 2026
Oct 26, 2026
Oct 25, 2026
- 李磊 A data company's long-term value depends on its ability to keep filling new gaps in models. 李磊 expects demand to persist. If data companies cannot keep producing high-quality data, models' own data-generation capabilities will drive …
- Xiaomi’s post-training livestream brought the training dynamics and costs of large-scale RL into public view. 李磊 says the team saw broad capability gains from large-scale RL early in training and wanted MiMo V2.6’s Pro and Flash models to give the …
Oct 20, 2026
- Andrej Radonjic Grass's advantage starts with getting AI systems onto websites that block data-center traffic. Andrej Radonjic says routing through residential IPs is the best way to get broad access, with browser infrastructure handling additional …
- Delphi Digital Frontier labs increasingly buy useful signal and speed alongside raw volume. A lab's discovery process can identify a small share of a dataset—for example, 0.16%—as useful, driving more targeted purchases. With new …
- Robotics creates demand for carefully selected behavioral video. Andrej updates Yan's roughly 1 billion figure to more than 100 billion indexed videos. Yan points out that indexing does not mean …
- Andrej Radonjic Training crawls are building the infrastructure for live inference and search. Andrej describes training-data collection as a “treasure hunt” that also maps the web. Grass has released its Content API under Live Context …
- Delphi Digital Yan recounts a founder's view that superior reinvestment returns can constitute a moat, and Andrej agrees. Yan cites the AI-data market estimate of $100 billion by 2030; Andrej sees expansion opportunities in synthetic traces, reinforcement …
Oct 19, 2026
Oct 17, 2026
Oct 14, 2026
- Nathan Lambert A lot of frontier progress in less-verifiable domains comes from high-quality data and environments, not simply “poof generalization.” Lambert says it is probably some of both, but a lot comes down to data. He balks at the scale of research budgets for environments, while …
- His new nonprofit Trillium will build open-source environments and share its research and model-use process — tasks, configs, trajectories, and failures — to demystify RSI. The name is the thesis: a trillium is “a spring ephemeral flower” that blooms before the canopy and helps spread nutrients. Lambert wants …
- SemiAnalysis Post-training, not a larger base model, drove GLM-5.3’s reported capability jump. Wenyao reports more than 10% improvement in agentic benchmarks; Kimbo calls the cybersecurity change from GLM-5.2 a “night-and-day …
- Kimbo Chen Z.ai’s edge may lie as much in research throughput as in new algorithms. Jordan’s pushback is that familiar post-training methods are not unique; the advantage can be running more, cheaper experiments on …
Oct 13, 2026
Oct 8, 2026
Oct 5, 2026
Sep 28, 2026
- 何允中 The industry’s center of gravity is shifting from human labeling to rubrics and RL environments. When 何允中 joined Scale AI, rubric data was still new; now, “rubric feels outdated—everyone is talking about RL environments.” He still …
- 孙一铀 The 2 guests openly disagree on whether rubrics have peaked, making the dispute a key signal for how much runway remains in the market. 何允中 believes the more accessible expert rubrics have largely been collected, especially as demand for static, single-turn or multi-turn text …
- 硅谷101 Benchmarks and data sales cannot be conflated. 孙一铀 says he has heard privately of data companies selling the same data they use for evaluation, and warns that commercial incentives cannot …
- In vertical markets, procurement is becoming a major bottleneck—and an opening for founders. A MOBA game with only a few thousand players can command RMB2M–3M for its source code; chip design has to solve commercial-software …
- 何允中的 contrarian take is that frontier labs may not have the right organizational DNA to solve data problems, and that a data vendor’s real moat is R&D. If a vendor both places an FDE inside a company and collects that company’s data, it may create a conflict of interest: enterprises “will not be entirely comfortable handing their data to you.” The right data vendor should operate like a software company, invest in R&D ahead of time, understand the industry’s problems, and then sell …
- 何允中 Fabricated data is harder to detect than contaminated data, and incentive design remains a vast unsolved research problem. Public-data contamination is relatively easy to identify, but an expert’s fabricated chemical simulation may take months to uncover; even …
