Summary

Robustness, not benchmark scores, is Bartolo’s standard for reasoning. Max Bartolo argues that reasoning should imply robust execution: if a model succeeds 999 times out of 1,000 but fails once on a simple task, that failure calls the reasoning claim into question. “If you had a calculator” that answered one elementary sum incorrectly, “you’d throw away your calculator.”
Evidence from model internals weakened Bartolo’s prior belief that apparent reasoning was mostly retrieval. Across roughly 40 factual and 40 reasoning queries, factual answers depended on a few documents containing the answer, while reasoning drew distributed influence from documents encoding relevant procedures. Carefully matched control questions produced a sharply different pattern, offering “a strong indication that the model was actually doing some form of reasoning.”
Human preference is an unsafe optimization target when style can masquerade as correctness. Annotators favored non-refusals, formatting, length and assertiveness, while factuality contributed surprisingly little to overall ratings. More assertive outputs were actually less factual yet judged more correct—an incentive structure that can reward style at the expense of correctness important to enterprise customers.
There may be no universal model ranking because preferences and use cases change with the user population. PRISM found models moving five or more places within a 21-model ranking depending on who asked and judged the questions; an early Command model also fell relatively for users in Asia. Bartolo’s alternative is inference-time personalization: one robust model conditioned on perhaps 100 evolving preference examples, avoiding the impossible economics of training “seven billion GPT-4s.”
Human-in-the-loop adversarial data appears to create unusually valuable training signal. People forced to defeat a model generate more diverse, implicit and realistic questions than ordinary annotators, and adversarial examples collected against weaker systems still improved stronger ones. Data from 2019 remained difficult enough to help evaluate Llama 3, which Bartolo says was still “quite far from human performance” on that robustness test.
Static benchmarks depreciate rapidly and can conceal both quantization damage and overstate progress. Bartolo cites saturation shrinking from roughly eight to 10 years for MNIST to one and a half to two years for SQuAD; once a task reaches 100%, the benchmark may be exhausted rather than the capability solved. The same incomplete evaluations can make a quantized 100B model attractive at roughly 35B per-token cost while missing losses in reasoning, long-range dependencies or long-context behavior.
Test-time compute becomes commercially interesting when users can buy the reliability appropriate to each task. Bartolo imagines a calibrated model answering when it is at least 60% confident for low-stakes work, but thinking longer and abstaining when a critical answer remains doubtful. He stresses that current models are “far from” this capability, making calibration, controllability and honest uncertainty more important than merely generating longer reasoning traces.

Deep dive

1. Pre-training appears to teach procedures, not merely store answers

Bartolo entered the influence-functions study with Laura Reese expecting models to retrieve from “a compressed version of all the information that they’re pre-trained on.” The results changed his view: factual responses leaned on a few documents containing the answer, while reasoning influence was distributed across many documents containing useful procedures.
The experiment covered roughly 40 factual questions and another 40 reasoning queries. Influence functions approximated how individual training examples affected behavior, revealing a much broader influence distribution for reasoning than for direct factual recall.
A matched control addressed the obvious similarity objection. Calculating the slope through points 2,2 and 3,3 required selecting and applying an equation; “The slope of the line is 1. What is the slope of the line?” looked lexically similar but merely repeated its answer. The two produced strongly contrasting influence patterns.
Scaling required approximating curvature rather than retraining without each example or directly computing an enormous inverse Hessian. More intriguingly, influential documents for the 7B and 35B models showed “very little correlation,” suggesting different model sizes may learn the same capability in different ways.

2. Reasoning earns its name only when execution survives probing

Bartolo calls reasoning “a bit of an overloaded term” and doubts that human-like internal cognition is the practical question. Next-token models already perform impressive tasks; what matters operationally is whether the behavior remains correct when inputs vary.
For him, correct reasoning should imply robust reasoning. If a model applies the slope calculation correctly 999 times out of 1,000 but inexplicably fails once, that failure brings the underlying reasoning claim into question.
Tim’s pushback — worth keeping: humans make errors, so why treat reasoning as binary? Bartolo’s answer is that machines do not tire and inherit software’s higher reliability standard: a calculator returning 17 for two plus seven would be discarded immediately.
Bartolo allows more error as task complexity rises, but not on grade-school mathematics. With GSM8K scores around 97%–98%, he suspects remaining misses may largely be test-set noise; the next test is letting humans devise fresh grade-school problems and probe whether the model can consistently do what it is asked.

3. Distributed adaptation could sit on top of centralized foundation models

Tim framed test-time training through ARC’s “transductive active fine-tuning”: adapt a shared model to one situation instead of relying exclusively on a globally fixed system. He asked whether that creates a more distributed AI paradigm alongside centralized providers such as Cohere.
Bartolo expects more local adaptation as expertise spreads. Cohere models such as Commander Plus or Commander could be downloaded from Hugging Face and further fine-tuned, although practitioners must avoid sacrificing useful general knowledge from the base model.
Language is the shared interface across people and models. If models help improve a common data resource, Bartolo argues that anyone with sufficient know-how and compute could train from it, broadening participation beyond the original model builder.
Tim’s software example made the interface shift concrete: he loads a Python multi-agent system into Open Interpreter, tells its agents to reveal their interfaces, then manipulates a database conversationally. No conventional UI is required; an intelligent agent can itself operate another intelligent system. Bartolo’s response: “Possibilities are endless.”

4. Alignment begins with an unresolved question: aligned to whom?

Bartolo treats safeguards as non-negotiable—AI should not damage the environment, worsen society or harm humanity—but says “what do we want to align AI systems to?” remains unresolved. Human wants differ across people and contexts, and society has not fully solved value alignment among humans either.
“Human Feedback Is Not a Gold Standard” challenged the single preference score used throughout RLHF. Pairwise judgments produced major early gains, but Bartolo says returns diminished and much of the gain reflected preferred output style, often evaluated by the same human preferences being optimized.
In one experiment, independent groups marked specific errors and rated overall quality from one to five. Refusals were heavily punished; formatting, style and length mattered strongly, while factuality ranked surprisingly low despite enterprise customers prioritizing correctness and non-repetition.
The sharper result came from prompting models to sound confident. As assertiveness rose from one to five, perceived error rates followed an elbow toward negligible levels—but a sample reviewed by “expert annotators,” with mostly Tom going through the data for factuality, found that assertive generations were less factual. Annotators nevertheless judged them more correct and did not realize they were being influenced.

5. Personalization may be cheaper and more individualized than one averaged preference

Tim asked whether granular feedback ends in thousands of attributes. Bartolo accepted the rabbit hole and proposed inference-time conditioning instead: perhaps 100 evolving data points could encode Tim’s preferences, allowing one robust model to adapt without fine-tuning a separate instance for every person.
Bartolo illustrated the subjectivity personally: when asked the sky’s color, he prefers “blue” over “the color of the sky is blue,” reflecting his culturally Mediterranean preference for directness. Most annotators may prefer otherwise; neither style is a universal gold standard.
PRISM, led by Hannah Kirk and awarded a Best Paper Award at NeurIPS, studied demographic, cultural, geographic and linguistic effects over roughly a year and a half. Even conversation topics varied: discussion of the Israel–Palestine conflict came disproportionately from participants based in the Middle East, affecting which future improvements their usage could drive.
Across 21 models, preferred rankings moved five or more positions depending on who interacted with them. An early generation of Cohere’s Command models performed strongly in most cases but dropped relatively for users in Asia—an actionable signal for improving whom a model actually serves.

6. Human-informed adversarial data can better represent real-world variation

Tim invoked non-robust features—such as odd blue pixels correlating with Mercedes—to distinguish correct outputs from correct reasons. Bartolo agreed that judging features in high-dimensional spaces is hard, but argued developers can control inputs and outputs to expose brittle behavior.
Synthetic perturbations such as Gaussian image noise or synonym substitutions often teach a model to defeat the noise-generating function, not real-world variation. Bartolo therefore favors human-informed adversarial collection: let people probe weaknesses, then turn their discoveries into training data.
His 2019 BDAI work compared BERT, RoBERTa and the older BiDAF system. Ordinary collection yielded bland questions like “What color is the sky?” Once a model answered that, annotators had to invent “What is the color of the thing in which the clouds are?”—more implicit, ambiguous and realistic language.
Data quality sometimes fell because questions became under-specified, yet diversity, complexity and robustness rose. Even examples collected against weaker models improved stronger models; question-answering startups reported especially large real-world gains after adding the dataset, beyond their benchmark improvements.

7. Robustness datasets can retain value across model generations

Bartolo’s strongest argument against pattern accumulation is open-ended human creativity: he can keep asking the same semantic question through constructions absent from training. “I shouldn’t be able to” confuse a system if its representation is genuinely robust.
Adversarial QA was later used to probe the Llama 3 family, where it is referred to as Dynabench QA. Although modern LLMs massively outperform 2019 systems on the task, Bartolo says they remain “quite far from human performance,” evidence that the test still exposes general weaknesses.
Dynabench operationalized an iterative loop: train on an initial dataset, ask humans to defeat the model, add those failures, retrain, then challenge annotators to move beyond the strengthened system. Each round raises the difficulty until fooling the model becomes exceptionally hard.
The platform supported work including hate-speech detection, sentiment analysis, PRISM and efforts with Common Crawl to improve pre-training crawls. It subsequently sat with the MLCommons community under its Data-Centric Machine Learning Research Working Group.

8. Benchmarks must evolve as quickly as the models they measure

Static benchmarks create a clear target for talented researchers, inviting Goodhart’s law and rapid saturation. Bartolo recalled roughly eight to 10 years for MNIST versus one and a half to two years for SQuAD, with the cycle shortening as the field grew.
Dynabench’s second purpose was dynamic evaluation. If today’s model scores 70% and a later one reaches 100%, Bartolo cautions that this does not establish universal competence; it may only show that an evaluation designed around yesterday’s failures has reached its limit.
Selecting a domain model because it leads MMLU or Chatbot Arena resembles deciding whether a PhD holder should perform surgery using a grade-school exam. Bartolo favors application-specific evaluations that evolve, potentially arranged like education: basic capabilities compose into increasingly specialized examinations.
DataPerf complemented that philosophy with data-centric challenges on Dynabench. Its premise was that community attention had swung heavily toward algorithms, architectures, hardware and resources, while improvements to training data still needed to remain central.

9. Cohere’s post-training cadence turned base models into products

Bartolo joined Cohere a few weeks before ChatGPT launched and built its post-training team. The base models were capable but followed corpus patterns rather than instructions: asked “What color is the sky?”, one might continue with “What color is the grass?” and “What color is the sun?”
Cohere built a rudimentary internal collection interface and sought 10,000 prompt-completion examples in under two weeks, a target employees exceeded. That data produced the first instruction-following Command generation and expanded into internal annotation, vendors and synthetic generation.
The team then delivered a new Command Nightly nearly every week for about 52 weeks. Four or five candidates were withheld because internal metrics showed no improvement or slight regression—users noticed even the missing releases and asked whether the model had changed.
Command R and Command R+ followed, releasing in April. Bartolo recalls Cohere ranking fourth among providers on Chatbot Arena, behind Anthropic, OpenAI and Google, while the team had already moved to its next generation.

10. Efficiency gains are only trustworthy to the extent evals are complete

Quantization can deliver an attractive near-term trade: a quantized 100B model could reach the per-token cost of a 35B model while probably performing better, and the 35B model can itself be compressed for further savings.
The hidden risk returns to evaluation. A small measured performance loss may conceal larger degradation in advanced reasoning, long-range dependencies or long-context use—dimensions that conventional test suites often fail to capture.
Bartolo sees no immediate evidence that connectionist architectures have reached a severe ceiling. His example of a fixable limitation is “glitch tokens”: tokenizer-training mismatch leaves some tokens rarely seen, while weight decay shrinks their embeddings until models struggle to distinguish their identities.
LLMs remain far less data-efficient than humans, consuming “many lifetimes” of text, but they can afford it when compute exists. They also gain a unique continuity advantage: copied weights reproduce the same compressed knowledge and capability, making continual pre-training potentially more efficient than repeatedly starting from scratch.

11. Long context is valuable infrastructure, not a reason to ingest everything

Cohere moved from 4K context in its first generation to 128K in current models, but Bartolo says maintaining high performance across the full window is challenging. Most user prompts are short; value appears in RAG, multi-document aggregation, tool use, persistent conversations and whole-codebase work.
Retrieval quality still matters, including embedding and reranking performance. Longer windows determine how much retrieved material a model can combine usefully, but every additional token brings processing cost even when caching mitigates it.
Persistent dialogue could also become personalization state: the conversation and accumulated feedback jointly encode how a user wants the system to behave. That creates an application opportunity beyond simply advertising a larger context number.
Bartolo rejects the claim that infinite context makes retrieval unnecessary. One could provide all existing data, but “do you want to?” Asking the sky’s color after requiring someone to read the entire internet is possible, inefficient and usually pointless.

12. Test-time compute should become a user-controlled reliability dial

Bartolo views ARC as another useful challenge, not an endpoint. Once solved, evaluation should move toward grounded questions: what AI should do, how it should operate in society and how existing capability can benefit humanity.
His team is exploring test-time compute more broadly than generating extra reasoning tokens. The product question is how a model communicates during deliberation and lets users customize the relationship among latency, compute, confidence and performance.
For a low-stakes task, a user might say, “If you’re at least 60% confident, give me the answer.” Proper calibration would mean that across 100 such cases, a model claiming 60% confidence succeeds about 60 times.
A critical task demands the opposite setting: “Think as much as you absolutely need to. If you’re in any doubt that the answer is wrong, tell me.” Bartolo’s honest qualification is that models are “far from” delivering this reliably, but he sees controllable compute and uncertainty as a central interaction problem for the coming years.

The State of AI: Humanoid Robots, AI Copyright Wars & China’s Growing Influence w/ Salim Ismail #157 Yamini Rangan, CEO @ HubSpot: How HubSpot Competes Against Salesforce