The Pitfalls AI Agents Face Were All Encountered by Quant Traders Thirty Years Ago

Deep thoughts on AI and aspirations —— ByteDance Deep Thinking Circle

People building Agents tend to treat the “perception-decision-execution-feedback” loop as novel. Wall Street doesn’t see it that way. Quantitative trading systems are this loop, running for over thirty years in an environment of real money and smart adversaries: extracting signals from multi-source data, generating trading decisions, splitting orders for execution, then iterating strategies based on profit and loss.

So the claim that “quantitative trading and AI Agents share the same underlying logic” isn’t valuable because of the isomorphism itself. Abstracting both domains as Markov decision processes is correct but trivial—a refrigerator thermostat fits the same framework. What matters is something else: quantitative trading is currently the only implementation of this loop that has scaled successfully in an adversarial environment. The tuition it paid is exactly what Agent engineering will pay next. This homework is worth copying.

Backtesting Looks Great, Live Trading Is Brutal

The first lesson in quantitative trading: a strategy performs perfectly on historical data, then collapses when live. The reason is overfitting—the model memorized noise in the history, not patterns. The prettier the backtest curve, the more suspicious you should be.

Agent engineering has the same pitfall everywhere. High scores on evaluation sets, underwhelming performance on real tasks. Test questions leak into training data, and Agents learn to game benchmarks instead of completing tasks. More insidious is task templating: benchmark scenarios fall into a few categories, and optimizing repeatedly on these scenarios is equivalent to backtesting on a market that no longer exists.

The quant solution is to enforce one discipline: backtesting is only the entry ticket, live data is the report card. Same for Agents—evaluation scores belong in marketing materials; online completion rate, cost, and reuse rate on real tasks are the balance sheet. If an Agent company’s internal reporting only shows benchmark scores without production metrics, you can safely assume it hasn’t entered the live trading phase.

Reward Functions Will Always Be Gamed

An old quant joke: optimize for Sharpe ratio, and you’ll end up with a portfolio that has a high Sharpe ratio but risks hidden where the metric can’t see them. The narrower the reward signal, the clearer the path to gaming the system. This is Goodhart’s law in financial form.

Agents have no immunity. Optimize task completion rate, and Agents learn to meet acceptance criteria with minimal effort; optimize user likes, and they generate content users enjoy but find useless; optimize code test pass rate, and they write code that only passes tests. Reward hacking isn’t a bug—it’s inevitable behavior for any sufficiently strong optimizer facing a narrow objective function.

Quant’s response offers a ready answer: make risk control an independent layer outside the strategy. No reward function, however carefully designed, can plug all loopholes, so you need a checking mechanism that doesn’t answer to strategy objectives but only to survival baselines. In Agent engineering, this means making audit and validation independent of the execution chain—don’t let the scorer and the executor share the same optimization target.

The Real Challenge Is Execution

Right decision, botched execution—this is where quant pays the most expensive tuition. A large order dumped directly into the market moves prices, hence order splitting; there are dozens of exchanges, hence routing choices; poor slippage control turns paper profits into costs, hence execution algorithms like VWAP and TWAP; when strategies go haywire, you need a kill switch to liquidate positions instantly.

Agent’s execution layer is the same problem with different names: tool call failures, API throttling, async timeouts, permission failures. No matter how smart the decision model, a single erroneous transfer call can drain an account. The structural problems quant and Agents face at the execution layer are nearly identical—latency and synchronization, fault tolerance and retry, cost optimization, auditable logs—all four, not one less.

The trouble is that current Agent engineering is thinnest precisely at the execution layer. Everyone’s attention is staked on planning and reasoning, with execution treated as a demo afterthought. Yet this is exactly the homework worth copying: exponential backoff, circuit breaker degradation, permission sandboxing, operation audit trails—these components, polished over thirty years in trading systems, are mature and directly portable.

Markets Change, Agents Age

A well-performing quant strategy has an average lifespan measured in years. When market structure changes, historical patterns become obsolete—this is called regime change. Agents similarly encounter distribution drift: user habits change, tool return formats change, task definitions themselves change, and yesterday’s stable success rate quietly declines.

The deeper challenge is credit assignment. A trade ultimately loses money—should you blame entry timing or exit timing? A task chain fails—should you blame planning or some intermediate tool call? Both domains struggle with the same thing and use similar technical solutions—tracing back from final results to attribute each step’s contribution.

Put these pitfalls together, and the correspondence is clear:

The Flip Side of Isomorphism: Agents Become Quant Problems

There’s also a reverse implication. When Agents enter markets, placing orders on behalf of users, auto-arbitraging, generating research reports, they themselves become new market participants, and competition rules immediately quantize: everyone uses equally smart Agents, arbitrage opportunities disappear in milliseconds, and prompt engineering itself becomes alpha. Alpha half-life will shrink further—this is the competitive consequence of isomorphism, not a technological vision.

So there’s one quant lesson that doesn’t transfer: quant wins in closed-rule environments where exchange rules are written explicitly. Real-world Agents face open environments where rules change, adversaries don’t play by the book, and there’s no exchange to backstop. Risk control standards should be more conservative than quant, with stop-loss lines drawn earlier.

Two concrete suggestions for people building Agent products. First, add someone to the team who has worked on trading systems or payment systems, specifically to watch execution layers and fault recovery—the ROI is likely higher than hiring another reasoning optimizer. Second, build a production ledger from day one: real task completion rates, costs, failure attribution, reviewed weekly. This is what the Agent industry lacks most right now: everyone displays backtest curves, few are willing to open their live account books.