Transformers Need Glasses! [Federico Barbero]

Summary

• Long context does not guarantee token-level fidelity: Federico Barbero argues that causal transformers eventually map distinct long sequences to representations too close for finite-precision hardware to separate. In the sharpest test, Gemini eventually output 1 instead of copying the final 0 from a long run of ones. A large context window alone therefore does not establish reliable copying at its edge.
• Causal attention contains a mechanical bias toward the beginning of a sequence, even though next-token training teaches models to favor recent tokens. Earlier tokens have more pathways through the lower-triangular attention topology; the final token survives only when it attends strongly to itself. Barbero’s synthesis of the U-shaped retrieval curve is concise: transformers are mechanically good at the start, learn to care about the end, and “the middle is somehow lost.”
• Quantization can turn gradual representational degradation into forced errors. Once two final-layer representations fall below the resolution of the chosen numerical format, aggressive quantization—including formats such as FP6 or 4-bit quantization—can make them indistinguishable and one answer must be wrong. That is a direct risk for inference: “copying is something that’s super fundamental,” including when a model must transfer an exact value into a tool.
• The counting failures look like heuristic pattern matching, not execution of an algorithm. Asked to sum repeated ones at lengths from 5 to 200, the model worked around 20, then produced values inconsistent with the increments, and eventually developed a strong mode at 100. Chain-of-thought decomposition into groups of five also diverged, supporting the cited “bag of heuristics” framing: training-fit shortcuts can generalize “horribly” out of distribution.
• Today’s large models may be using substantial capacity to counteract attention’s contraction. Residual connections, MLP expansion, windowed attention, diagonal heads, and heads attending to a near-zero beginning-of-sequence token can preserve information or implement costly no-ops. Barbero sees both an architectural clue and a compression opportunity in asking why models learn “quadratic no-ops.”
• Formal expressivity claims do not remove the practical limitation because many results assume hard or average-hard attention, while some allow memory or precision to grow with sequence length. Infinite precision would avoid this particular representational collapse; deployed precision “is what it is.” Barbero emphasizes his measurable quantity—the distance between actual internal representations—rather than only constructive proofs that an idealized transformer can represent a language.
• Barbero ultimately favors a hybrid system whose language model orchestrates specialized components, while questioning a clean boundary between reasoning and memorized heuristics. A replaceable math, chess, vision, or even RNN-based counting unit could supply capabilities a transformer lacks, although combining the systems remains extremely difficult. His deeper criterion is practical: machines should exploit being computers, avoid trivial arithmetic and conceptual mistakes, and use learned compression to form concepts from which apparently creative leaps emerge.

Deep dive

1. A single late token can disappear inside a long context

• Barbero’s construction compares growing sequences whose correct answers differ only because one contains an extra final token. As length increases, the last-layer, last-token representations converge until finite precision maps them to the same value; “one of them has to contain a mistake.”

• The decisive example is copying, not difficult arithmetic: give the model a long sequence of ones followed by zero and ask for the last element. A human need only look, yet Gemini eventually outputs 1 instead of copying the final 0. Unlike counting, this should be a trivially generalizable operation.

• The paper measures the norm between the two final representations and studies it becoming arbitrarily close to epsilon. Barbero values that diagnostic because it identifies a quantity inside a real transformer that tracks failure, rather than merely demonstrating that an abstract architecture could represent the task.

2. Causal topology privileges the beginning while training privileges the end

• Causal masking makes attention lower-triangular: each position can look backward but not forward. It enables efficient autoregressive training, unlike bidirectional systems such as BERT, but Barbero argues that this engineering choice also determines how information can propagate.

• An early token has many possible pathways through layers to reach the prediction position. The final token has essentially one preservation route—attention to itself—so keeping it strong consumes attention mass that could otherwise read the context: “If you want the last token to survive, you can’t really have it attend to all the other tokens as well.”

• The host’s recency objection is important: deployed models often remember recent material better than old material. Barbero separates mechanics from training dynamics—the topology retains the beginning, while next-token prediction usually rewards nearby context and trains attention toward the end.

• That conflict offers a mechanism for “lost in the middle.” Models are structurally advantaged at the beginning and behaviorally optimized for the end; the middle gets neither advantage, producing the observed U-shaped retrieval profile.

3. Graph theory turns information loss into a tractable mechanism

• Barbero’s route into the problem came from graph neural networks, where information propagation can be related to spectral graph theory, random walks, and heat diffusion. Graph convolutional networks can be viewed as discretized heat equations over graphs, making sensitivity between nodes mathematically analyzable.

• The useful quantity is commute time: starting from one node, how many random-walk steps are expected before reaching another and returning? Changing a graph’s connectivity changes that time and therefore how easily a graph network can make two distant pieces of information communicate.

• Causal transformers have enough topology to import this machinery. Under simplifying assumptions that omit nonlinear contributions, repeatedly applying triangular attention matrices leaves one eigenvalue at 1 while the others decay below 1; in the limit, the surviving information points toward the initial token.

• Barbero also cites a security paper in which repeated tokens eventually make the output become the beginning-of-sequence token. He sees this as consistent with the same mechanism, while acknowledging that the theoretical analysis uses broad simplifications.

4. Long-context models survive through expansion, no-ops, and selective attention

• The host’s strongest challenge is empirical: models now accept contexts around 2 million tokens without obvious total collapse. Barbero’s honest answer is that “someone has to open up these models and figure this out”; spectrally, attention alone should contract, but the full system contains countervailing mechanisms.

• Residual connections preserve earlier states, while MLPs can expand what attention has weakened. If attention copies 100 items at roughly 1/100 strength each, an MLP can multiply the result back up—although layer normalization complicates this clean contraction-and-expansion picture.

• Longer-context architectures may also use windowed or alternating attention, effectively limiting how much information must mix at once. Barbero thinks the fundamental problem is fitting large amounts of information into finite bits, so a possible solution is to compress information and make the model care about fewer things.

• The host points to Llama and Gemma, where many heads appear to implement diagonal no-ops or attend to a beginning-of-sequence representation with norm near zero. Barbero says Razvan Pascanu would likely frame the large number of heads as exploratory diversity: many heads offer routes at initialization, after which unnecessary ones shut down. Barbero sees potential compression savings in these “expensive no-ops.”

5. Precision, tokenization, and prompting change when the failure arrives

• Numerical precision supplies the hard boundary. Heavy quantization can merge sequences that remained distinguishable at higher precision, converting a soft loss of fidelity into an unavoidable wrong answer. The host, citing Roon’s claim, presses the implication that frontier models already struggle beyond roughly 100 copying or counting elements before being reduced to 4-bit precision.

• The convergence rate is input-sensitive. Spaces can determine whether the tokenizer sees a new token, sequence entropy matters, and even particular digits—Barbero noted that larger digits such as nine tend to produce larger magnitudes—can alter how quickly two representations collapse.

• Interleaving additional zeros into a ones-heavy sequence improves recovery of a final zero because the added zeros counteract the ones dominating the representation. More revealingly, copying the first token seemed much easier than copying the last, matching the predicted asymmetry in path counts and partial derivatives.

• Chain of thought did not restore an algorithm. Breaking repeated addition into groups of five still drifted into nonsense; across sums of 5 through 200 ones, outputs eventually clustered at 100. Barbero’s inference is hedged but pointed: the model is “probably not mechanically counting”—100 is merely a plausible learned answer for a large sum.

6. Expressive in theory is not reliable in finite-precision deployment

• Many formal Transformer expressivity constructions assume hard attention—weights exactly zero or one—with some relaxing this to average-hard attention, which distributes weight (1/k) over a set of (k) tokens. Other constructions let memory or precision grow with sequence length; roughly, usable memory scales with hidden dimension, floating-point precision, and token count.

• Those assumptions explain why infinite-precision RNN results can coexist with collapse in real systems. With infinite precision, arbitrarily close states remain distinguishable; on hardware, “precision is what it is,” so a representational distance eventually crosses a concrete threshold.

• The host also notes a conjecture around RASP: short programs composed of Transformer-compatible operations may be easier for models to learn and generalize. Barbero treats that constructive approach as complementary, but emphasizes that his result offers something operational: measure the internal distance and watch the failure mechanism occur.

• Barbero also points to evidence that merely reordering chains of first-order-logic implications can sharply change whether a model succeeds. He therefore favors hybrid systems—a fixed language-model orchestrator with replaceable chess, math, vision, or RNN-based modules—but says the difficulty of designing their interaction is likely why such combinations are not used very often.

7. Reasoning may be compression plus search, not a cleanly separable faculty

• Barbero calls reasoning “very ill defined.” A correct computer program can generalize arbitrarily far and be formally verified, yet whether that constitutes reasoning is unclear; conversely, even humans who know how to sort an array will probably err on an array with a million entries. His ideal system would avoid both trivial arithmetic and conceptual mistakes while leveraging machine strengths.

• The host proposes intelligence as knowledge-acquisition efficiency and contrasts heuristic chess engines with AlphaZero’s novelty. Barbero resists the distinction: both conduct tree search with different evaluation heuristics, and AlphaZero’s creativity may simply reflect guidance “not aligned with what humans necessarily think.”

• His chess example makes compression concrete. Grandmasters reconstruct real-game positions far better than amateurs but lose that advantage on random arrangements; familiar structures—such as pawns on f7, g6, and h7, a bishop on g7, and castling—become one chunk rather than several independent facts.

• Inspiration then arrives only after that compressed representation has formed. Focus, sleep, health, and sustained investment are conditions he names for occasional moments when “everything clicks,” with each step accumulating on the last. The host’s closing phrase, “epistemic foraging,” captures the idea that prolonged experience enlarges the usable cloud of knowledge.