Why a 4-Billion-Point Kaggle Dataset Still Couldn’t Beat Chance — Ian Quigley, Leash Bio

Summary

• Leash’s reported advantage is inexpensive, carefully characterized experimental data rather than an elaborate model architecture. RJ Honicky describes the shop as a ten-person operation; Ian Quigley says Leash has developed about 2,000 protein constructs and run roughly 5,000 screens, each covering at least 6.5 million molecules. A one-off CRO screen might cost $100,000, depending on the handholding. Quigley’s defensibility argument is operational: “there’s just so many things where it can go sideways.”
• Leash’s Kaggle competition exposed how quickly predictive performance disappears when chemistry genuinely changes. Its roughly 4-billion-interaction release dwarfed the then-largest aggregated binding database, but none of approximately 2,000 competing teams beat chance on the most distant chemical library. Quigley found that encouraging rather than discouraging: “wow, this is much harder than we thought” strengthened the case for collecting better, broader experimental data.
• Chemical benchmarks may reward recognition of scientists’ habits rather than understanding of physics. A classifier identified the correct chemist among its top five guesses with 60% accuracy across 1,815 chemists; representing molecules only through author probabilities predicted ChEMBL activity about as well as chemical representations. Combining the two did not meaningfully improve results—evidence, Quigley argues, of models “learning a lot of sociology.”
• Hermes demonstrates some external transfer, while its failure modes still shape Leash’s experiments. Trained exclusively on internally generated data, the simple embedding-and-cross-attention model performed about as well as Boltz on binding evaluations Quigley estimates covered 13 million tests and roughly 500 targets. Quigley interpreted attention maps as showing protein “nicknames” rather than functional understanding; engineered mutations broadened the model’s attention, but the scaling thesis remains “We hope so.”
• The newer opportunity is a physically grounded reasoning sandbox operating at database speed. Ian describes agents starting with ten binders and ten nearby non-binders, then choosing molecules to test their hypotheses. RJ Honicky says the agents get a 100-molecule lookup budget and find the best binders within a measured 6.5-million-molecule library. The experiments have already happened—“like it’s a coding sandbox”—making this early evidence inside a finite measured space, not unrestricted drug design.
• Binding predictions remain constrained by chemical coverage, DNA effects and incomplete potency calibration. Off-DNA resynthesis works for about 90% of easier hits and 50% of harder ones in Leash’s hands, while sequencing readouts support relative rankings more readily than reliable binding constants. Asked whether current chemical coverage is enough, Quigley answers: “Not remotely.”
• Mouse results support a partnering strategy, not a clinical-efficacy claim. For a few series, Leash moved from library hits to good mouse PK/PD in six months, including inflammation-related cytokine changes consistent with the intended targets. Quigley describes potentially selective starting material for about 700 targets and partner discussions around new entities—“little bits of gold”—but presents no human results or firm clinical timeline.
• The broad AI upside depends on transferring knowledge beyond molecules that cheap screening already finds. Quigley concedes the Kosuri paradox for an already-screened target: “you don’t need a model for the thing.” The hoped-for payoff is faster optimization and transferable chemical reasoning, while the hosts’ economic challenge remains: discovery may cost only a few million dollars against $200 million–$1 billion for a successful drug’s development.

Deep dive

1. Small molecules offer reach but complicate safety

• Leash measures small molecules binding to proteins, trains models on those measurements, and moves selected chemical material toward partners’ drug programs. RJ Honicky describes the shop as a ten-person operation; Quigley says Leash has developed about 2,000 protein constructs and run roughly 5,000 screens, each covering at least 6.5 million molecules. A one-off screen at a CRO, he estimates, could cost $100,000, depending on the handholding.
• The therapeutic attraction is reach and convenience: small molecules can become pills and enter cells, whereas larger antibodies and protein therapies struggle with oral delivery and intracellular access. Quigley thinks protein-drug design has advanced faster because sufficient data exist; comparable predictive capabilities for small molecules are “really not here.”
• RJ’s counterweight: going everywhere is “good and bad.” Cell penetration, tumor access and potentially easier blood–brain barrier passage also create opportunities for toxicity. Quigley names liver problems and heart arrhythmias; changing a molecule to remove one liability can compromise its intended activity. Screening potential off-target proteins is one way Leash hopes to anticipate these problems.
• The coverage ambition is explicit: “it is our aspiration to have screened every protein,” extending beyond humans. Species differences could inform whether mouse or rat experiments are more appropriate. Cellular behaviors such as membrane crossing sit another biological layer away; Quigley hopes biophysically informed models might eventually be fine-tuned on smaller datasets for those tasks.

2. DNA encoding turns pooled chemistry into sequencing data

• Quigley’s DEL explanation starts with a fishhook: attach protein to a magnetic bead, immerse it in pooled chemicals carrying DNA barcodes, retrieve the bead, wash it, and sequence the attached molecules’ barcodes. Unlike testing one chemical per tube or well, pooling removes much of the logistical overhead of screening large collections.
• Combinatorial construction supplies the scale. Thousands or tens of thousands of building blocks can produce millions of combinations. That restricts chemistry, but also creates useful comparisons: many molecules differ in only one component, making it easier to identify which part drives binding rather than merely observing that an entire molecule works.
• Conventional, individually arrayed libraries top out in the low single-digit millions, Quigley says; the largest DEL produced contained 40 trillion molecules. He cautions that libraries encounter problems at that size.
• Leash’s measurement philosophy is deliberately narrow: “Sequencing DNA is another way of taking pictures.” Quigley sees imaging and sequencing as the genuinely scalable readouts for biology-plus-ML, while acknowledging proteomics researchers may disagree. Falling sequencing costs directly influence Leash’s collection strategy; competition among newer sequencing companies has restarted price declines after a period of relative stagnation.

3. Barcode controls distinguish chemistry from assay artifacts

• The enrichment measurement needs comparisons, not just recovered DNA. Quigley describes negative controls for binding to the bead itself and for broad, nonspecific protein stickiness. PCR introduces another distortion because some sequences amplify faster; unique sequence tags let Leash collapse amplified copies back toward the original observed events.
• DNA is also an active participant: many proteins naturally bind it. Leash therefore uses 32 or 48 different barcodes encoding the same small molecule. If one encoding enriches much more strongly than the others, the signal may come from DNA recognition rather than the chemical. Sometimes the enriched sequence recovers a transcription factor’s known binding motif—an artifact that also provides evidence the protein was made correctly.
• Pooled synthesis can react imperfectly, leaving a mismatch between a barcode’s intended chemical and what was actually produced. Leash builds manufacturing controls around that risk. Quigley rejects the assumption that individually arrayed compounds are automatically cleaner: he says studies find their labels “typically off by twenty to thirty percent,” which he thinks is comparable to DELs, “maybe worse.”

4. On-DNA measurements and off-DNA drugs answer different questions

• A host presses the bulky-tag objection: DNA hanging from a molecule could obstruct a pocket and create a false negative. Quigley’s rebuttal depends on the modeled question: “we know it’s there, and we don’t pretend that it’s not.” Leash includes a placeholder marking DNA’s attachment position rather than asking the model to treat a tagged compound as untagged.
• A drug program still requires removing that dependency. Leash manufactures interesting chemicals off DNA and tests them in functional assays; Quigley reports that about 90% work for easier cases and about 50% for harder ones, in his team’s hands. He explicitly acknowledges other groups have different experiences and that removing the barcode can change behavior.
• For training, the narrower claim is reproducible physical signal. Fresh protein batches and newly manufactured libraries, tested across many months, recover mostly the same hits in roughly the same order. That makes the tagged system useful even though it is not identical to the eventual medicine.

5. Screening breadth changes the incentives behind DEL experiments

• Historically, Quigley says, expensive sequencing encouraged treating a DEL screen as a major undertaking, often reserved for unusually difficult targets after other methods failed. Teams might spend two to six months producing and characterizing protein. Maximizing the chance of finding any usable hit then favored very large chemical libraries.
• Leash instead screens each library against many hundreds of targets, learning the library’s behavior rather than judging it from one difficult protein. One supposedly identical library behaved differently across manufacturing runs; only broad screening exposed the recurring motifs. Understanding those chemical patterns also helps troubleshoot protein production, making the assay itself an accumulating source of diagnostic information.
• Quigley and co-founder Andrew Blevins brought genomics-methods experience and years of Recursion mistakes to the problem. Their instinct is to hunt for cheating opportunities in noisy measurements—for example, whether a model can predict which instrument produced the data. “We had this real eye” for sources of noise that need chasing down.
• Brandon Anderson’s reversal is worth retaining: he once told a Leash angel investor, “that’s money you’ll never see again.” On the podcast: “I was wrong.”

6. Kaggle separated familiar chemistry from genuine generalization

• Leash’s competition release contained about 4 billion physical interactions, measured by one person in Quigley’s basement apartment. At the time, roughly two years earlier, BindingDB’s aggregated protein–small-molecule collection held a little under 3 million. The main competition library, supplied through AlphaMa in Shanghai, contained about 130 million molecules sharing a central triazine hub with three spokes.
• The easiest evaluation allowed one shared building block between training and test molecules, with the other spokes different. A harder split shared no building blocks but retained the triazine hub. These were designed to offer attainable wins as well as progressively harder tests, rather than presenting competitors with a single random train–test split.
• The hardest evaluation used a different Leash-designed library: no common hub, no shared building blocks, different attachment chemistry. That library was withheld from the public evaluation. Competitors were not told the three-part difficulty structure in advance.
• After about 2,000 teams competed, results were acceptable on shared-building-block chemistry and sometimes improved on the shared-hub tier, depending on the target. On the distant library, “no one did better than chance.” Even the winner failed that test; three of the top five were “the noisy tutorial notebook.”

7. The response favored protein breadth and better measurement

• Quigley admits Leash assigned about a 5% chance that contemporary co-folding advances would make the company unnecessary. Instead, the difficulty felt validating: “we are extremely bitter lesson pilled over here.” If scalable measurement could supply reliable signal, this looked like the moment to “pour gas on this.”
• The budget decision was not simply “more molecules.” Kaggle covered three well-studied proteins, reinforcing the need for many more targets. Better characterization of a smaller library looked more useful than the approximately 100-million-compound collection. Quigley contrasts three proteins against a billion chemicals with thousands of proteins against single-digit millions: variation matters on both axes.
• Molecular representation was another useful result. Basic SMILES strings “kind of beat everything else,” although not by a “resounding thumping.” SMILES encode a graph traversal as text, not intrinsic 3-D coordinates; more elaborate representations did not justify switching.
• Skeptics asked whether any strong competitors had participated. Leash responded by releasing everything—raw read counts, binary labels and evaluations—through Polaris, allowing independent processing and testing. Quigley’s invitation remains: “if anyone out there thinks they’re awesome, then, like, please show us,” because the company wants the problem solved.

8. Hermes pairs simple architecture with demanding baselines

• Hermes was built to iterate quickly on training regimes, not initially to be a state-of-the-art model. It combines an ESM protein embedding with a ChemBERTa molecular embedding, joins them through cross-attention, and predicts a binary binding label. Quigley calls the architecture “incredibly simple”; ChemBERTa supplies a SMILES-token-based representation.
• RJ points out that an MLP would be simpler still. Quigley explains that earlier work had found some graph models retained performance after positional coordinates were scrambled. Brandon connects this to the “Picasso problem”: recognizing local features without understanding their arrangement. Cross-attention offers an opportunity—not a guarantee—for molecular features to interact meaningfully with protein residues.
• Evaluation therefore separates increasingly distant chemistry. Holding out building-block identities is insufficient when near-identical blocks remain on both sides. Leash clusters building blocks by similarity and prevents whole clusters from overlapping between training and test. Performance drops under that stricter regime, exposing how easy apparent generalization can otherwise be.
• XGBoost is the recurring sanity check. Its performance declines as shared building blocks go from two to one to none, then falls further with cluster-based separation. Hermes is “pretty steady” by comparison across these regimes. The lesson is not that simple baselines are weak: “you’ve really got to get pretty far out of distribution for XGBoost to really tank.”

9. A chemist’s signature can masquerade as chemical understanding

• The authorship project began when a medicinal chemist walked past Quigley and Blevins at Recursion and remarked, “Hey, looks like a Stuart Schreiber molecule.” Recognizing the creator from the structure alone haunted Blevins for six or seven years: could a model identify the scientist and infer their intent?
• Leash collected ChEMBL molecules from 1,815 chemists, using what Quigley recalls as a cutoff of more than 30 papers and 600 molecules. A classifier achieved 60% top-five accuracy at predicting authorship. Confusion often involved chemists who shared publications and institutions, consistent with distinctive styles and trainees inheriting some of their mentors’ preferences.
• The stronger test removed explicit chemical representations from the activity predictor. Molecules became roughly 1,800-element vectors of creator probabilities. Those vectors predicted ChEMBL activity about as well as chemical representations; combining the two did not meaningfully improve performance. Quigley’s interpretation: “It’s not learning ground truth,” but scientists’ choices and fingerprints.
• This is not an accusation that chemists are doing their jobs badly. They aim to reach drugs with as few chemical series as possible, not generate unbiased training corpora. Unrecorded decisions about cost, feasibility, available building blocks and expected failure shape what gets made. His warning: “when you train models on the, on the exhaust, you might be in trouble.”

10. Clever Hans explains why impressive benchmarks can mislead

• A preferred building block can become a sociological feature. Quigley describes chemists reusing something that worked in a prior clinical program or seemed helpful for solubility. The model may learn the scientist’s confidence and habits rather than the physical reason a new molecule should bind.
• Brandon’s concrete shortcut is molecular weight. Chemists find a hit, add groups to improve binding, then add other groups to fix properties such as solubility; successful molecules accumulate elaboration. Quigley recalls a weight-based baseline tying for first at CASF-2016, or finishing very close to second. RJ stresses that this is a dataset property, not an intrinsic law of chemistry.
• Quigley’s historical analogy is Clever Hans, the apparently mathematical horse. It tapped out answers successfully until questioned by someone who did not know the answer. The horse was reading posture and microexpressions, not arithmetic. Likewise, a chemical model can appear competent while reading the creators’ faces, metaphorically, rather than solving the underlying problem.

11. External validation shows transfer without settling the physics question

• Because Hermes trains exclusively on internally generated measurements, Leash can use external collections as evaluations rather than training material. Quigley estimates the paper covered around 13 million tests across roughly 500 protein targets. On the binding-prediction task, Hermes did about as well as Boltz, then a leading structural model; he emphasizes that Boltz can do many things Hermes cannot.
• Brandon asks whether Boltz had trained on evaluation data. Quigley says some training-subset boundaries were unclear. Many evaluations came from the Boltz team, with Leash adding more molecules. Relative strengths differed by protein family, which he suspects reflects training distributions—or some memorization—rather than a universally superior architecture.
• BRD4 supplied a revealing case: an orthogonal library included an identical molecule already represented with that protein in a PDB crystal structure. Hermes performed poorly on the distant chemistry; Boltz was perfect. Quigley suspects Boltz had already seen that exact protein–molecule structure. “That checks out” is his conclusion, not a claim that the comparison proves either model understands physics.
• Asked whether this validates broader scaling, Quigley answers, “We hope so.” The next emphasis is intentionally chosen activity cliffs: nearly identical molecules or proteins with sharply different behavior. His molecular mnemonic is the “magic methyl”—“kind of mythical,” he cautions—where one tiny thorn’s presence, absence or position can determine whether a molecule works.

12. Mutant screens manufacture the missing hard negatives

• Quigley says current co-folding models can fail a direct pocket test: digitally fill a binding pocket “like you’re filling it with cement,” remove it, or enlarge it, and the model may still place the molecule where the pocket used to be. Structural datasets lack strong negatives; successful protein–molecule complexes do not teach every way an interaction can be broken.
• Leash tests this physically. First screen an intact protein; then add a high concentration of a known pocket-binding compound and rescreen to see which chemical series it displaces. Next mutate an amino-acid residue in that pocket and screen again. If the same series disappear, the two interventions provide converging evidence that those chemicals use the identified pocket.
• The obvious objection is global protein damage. Quigley’s control is that other chemical series, binding elsewhere, remain intact: the mutation did not simply destroy every interaction. Leash runs many hundreds of these screens and can screen about 100 protein constructs on a big week. Mutations can also improve binding or create opportunities for different chemicals.
• The displayed plot makes that last point concrete: one mutation improves binding for colored populations of related molecules, while a second mutation expels those populations entirely. Because DELs contain hundreds of close relatives across several chemical series, repeated population shifts can connect one small-molecule side group to one protein residue much more convincingly than an isolated hit disappearing.

13. Attention maps caught Hermes inventing protein nicknames

• Leash expected Hermes to attend to binding pockets, or perhaps broadly across conserved functional domains. Instead, it focused on short, non-conserved stretches away from the functional regions—distinctive sequences that could identify a protein without explaining its behavior. Quigley’s interpretation was that Hermes had invented “a nickname for the protein”: remember ten or twelve distinguishing amino acids, then retrieve the associated binding behavior.
• Engineered mutants undermine that shortcut because almost identical proteins now have substantially different outcomes. After adding mutant data and retraining, Leash saw attention distributed much more evenly. The intended lesson extends beyond any one gene: the model should discover that the nickname strategy no longer works reliably, rather than simply memorize another identifier.

14. Language agents turn measured chemistry into compute-speed experiments

• Quigley’s newer direction uses readable English to represent molecules, proteins and behavior. With open-source language models including Qwen and GLM, Leash teaches structure descriptions, differences between molecules, and explanations of binders versus hard negatives. The growing activity-cliff corpus supplies reasoning traces grounded in reproducible physical changes. An agent starts with ten binders and ten nearby non-binders, then proposes why the binders work.
• It can select five or ten molecules to test that hypothesis, receive results, and revise its explanation. These are database lookups against experiments already performed: “It will get back the answer instantly, like it’s a coding sandbox.” RJ Honicky says current agents get a 100-molecule budget and find the best binders among 6.5 million. Quigley describes the approach as encouraging early signal.
• The scale comparison is a medicinal chemist’s structure–activity relationship campaign: a person might see a couple dozen, perhaps 50, in a career; Leash extracted over 10,000 “last weekend.” Natural language “seems like” a useful regularizer, Quigley argues, because adding parameters to Hermes could let it memorize mutants too. He thinks the reasoning approach is “much more resistant to the memorization problem.”
• RJ presses why avoiding shortcuts should also improve answers. Quigley points to rapid rewards and choices humans would not make: “It’s gotta have the counterfactuals. It’s gotta make mistakes.” Pooled measurements provide those alternatives without recording only a chemist’s preferred path. The sandbox remains restricted to screened chemistry; ordering new catalog molecules and retesting them is a future extension.

15. Chemical coverage and potency remain unfinished work

• RJ asks whether DELs favor flat molecules. “It depends on the building blocks,” Quigley replies. Leash uses forked, Y-shaped designs, linear designs and macrocycle-like geometries. DNA placement, available building blocks and compatible assembly chemistry all constrain the result; a library can be good for one target class and poor for another.
• Brandon challenges whether 6.5 million molecules matter in chemical spaces often described as (10^{40}) or (10^{60}). Quigley points to transfer into external ChEMBL chemistry, but does not call current coverage sufficient: “Not remotely.” The roadmap balances coarse diversity for new target shapes against fine-grained changes that expose activity cliffs.
• Building-block availability is a real limitation: moving a nitrogen around every position of a ring may require custom synthesis. Quigley is less worried about fragile attachment chemistry. Over roughly the past four years, DNA-compatible methods have expanded beyond digestible amide bonds to carbon–carbon and carbon–nitrogen bonds.
• Relative potency is more accessible than absolute binding constants. Repeated screens recover similar rankings, but reads are not directly reliable (K_D) values. Leash is adding SPR biophysical measurements and exploring a potency ladder for calibration; “I wouldn’t say that we’re there yet.” Unexpected DNA binding by proteins not previously described as DNA binders further complicates translation.

16. Mouse results feed partnering rather than a clinical timeline

• For a few chemical series, Quigley says Leash progressed from library hits to good mouse PK/PD in six months with few medicinal-chemistry cycles. One inflammation compound changed blood cytokines as expected from its targets. That describes exposure and biomarkers of intended activity; the episode does not present a human efficacy result.
• Asked what enabled the speed, Quigley credits, in part, starting with drug-like chemistry. Library design uses a Thompson-sampling-like approach: generate possible libraries, score them against an optimization function, and choose a smaller collection. Objectives include desired binding, avoiding other targets and predicted ADME properties, rather than maximizing library size alone.
• Brandon’s “When people?” gets a process answer, not a date. Leash is discussing partnerships and new entities to advance selected programs. Declaring a development candidate, establishing manufacturing and completing additional safety work still take time; a fast discovery engine does not remove those subsequent requirements.
• The byproducts can themselves have value: “No one wants to sell reasoning traces about, like, political theory,” but Leash owns chemical starting material. Quigley thinks it has material that “looks pretty selective to us” for about 700 targets. These are opportunities with different economic value, not programs it intends to pursue simultaneously—“little bits of gold” generated along the way.

17. Discovery economics sharpen the case for transferable models

• A host’s challenge is that discovery is a small fraction of drug development: perhaps a couple million dollars against $200 million–$1 billion for the steps leading to a successful drug, depending on the indication and testing requirements. Even allowing for failed candidates, is Leash solving the economically important bottleneck?
• Quigley argues some early problems are valuable enough. Cytokine targets may have strong genetic evidence and successful antibody therapies yet remain hard to drug with small molecules. A pill replacing an infusion, without refrigerated transport, would be desirable. He says Leash has selective material against several such cytokines, while acknowledging substantial downstream capital requirements.
• He also concedes a “lamppost problem”: scalable measurement currently favors early biology, not millions of mice or clinical trials. A host suggests an eventual oracle would need mouse and clinical data; Quigley answers “maybe.” For a trusted target, sufficiently good ADME and off-target predictions might already justify going directly to a mouse study without requiring clinical data first.
• Brandon’s “Kosuri paradox” asks why models are necessary if cheap experiments already deliver the molecules. For one screened target, Quigley concedes, “you don’t need a model for the thing.” The hoped-for payoff is faster optimization and transferable reasoning into unscreened targets, agriculture, veterinary science or materials. Only if chemical understanding generalized broadly enough would further screening eventually become unnecessary.

18. Process discipline is the moat, and purification is the next bottleneck

• Asked what prevents a better-funded lab from outscaling Leash, Quigley points to accumulated experimental mistakes and the culture required to detect them. Teams need frictionless metadata tools, historical comparisons and persistent batch-effect investigation. Leash rebuilt selection and much of protein expression, then added quality proxies. His claim is that replication would take time, not that more capital could never compete.
• His best example is lumacaftor-like chemistry. Compounds resembling the cystic-fibrosis drug repeatedly hit certain protein preparations; investigating expression conditions suggested those hits tracked misfolding. Leash now treats that pattern as one warning: “that protein probably didn’t fold right.” Such data may be excluded from training and the experiment rerun under friendlier conditions. The diagnostic emerged from noticing an unexpected recurring chemical series.
• Quigley welcomes sequencing costs “drop like a rock”: “Cheaper sequencing, more molecules” directly serves the desire for more chemicals and more measurements. He treats the trend as something already helping, not an unsolved hypothetical.
• His more concrete wish is massively parallel small-molecule purification. HPLC and similar cleanup approaches do not deliver sufficiently clean compounds at the desired scale. Easier purification would let Leash take large numbers of DEL hits into different assays and “orthogonally validate all those measurements”—expanding confidence beyond the platform’s deliberately unusual primary readout.