Why backtests lie

October 14, 2026·7 min read

Fitting the past too closely (overfitting) and testing too many ideas (the multiple-comparisons problem) explain why the strategy that worked on historical data usually did not.

Somewhere right now, someone is staring at a backtest that shows 34% annualized returns. The chart climbs like a staircase. The Sharpe ratio reads 2.1. That person believes it. That person should not.

A backtest is a test of a trading strategy on historical price data. You run your rules against old prices and see how much money you would have made. The result looks like proof. It is only a record of the past, and the past is easy to fit.

Annualized returns (yearly returns) state the yearly rate that would produce the final result if it held steady. The Sharpe ratio (return compared with risk) measures how much return a strategy earned for each unit of risk it took. Both numbers look exact. Both can mislead. Exact numbers feel like proof. They are only a record of the past.

The catch is that a strategy worked on data the researcher already saw. Without strict discipline, that result proves nothing at all. Two traps explain most of the damage. Here they are.

Overfitting, Concretely

A strategy with many adjustable settings can fit any past data, even data with no pattern at all. Each adjustable setting is a parameter. Think of a parameter as a dial you can twist. Twist enough dials, and you can make any curve line up with any past.

Three settings are common. The lookback window is how far back the strategy looks. The entry threshold is the signal that triggers a buy. The stop-loss distance is how far a price must fall before the strategy sells. Each parameter is a degree of freedom (one more way to bend the strategy to fit the past).

With enough settings, you can fit any historical dataset perfectly, including pure noise. A perfect fit to the past is not a prediction. It is a description of what already happened. Noise is random movement with no pattern. A signal is a real pattern that repeats.

Overfitting means the strategy learned random movement, not a real pattern. Picture a student who memorizes last year's exam answers instead of learning the subject. That student aces the old exam. Then the teacher hands out a new exam, and the student fails. An overfit strategy does the same thing. It memorizes the past and fails the future.

This is not a metaphor. Take a random walk (a price path driven by chance, with no pattern). Imagine flipping a coin to decide whether the price steps up or down each day. No skill. No pattern. Just chance. A researcher who tries a few dozen setting combinations on that coin-flip data will routinely discover a strategy with an impressive Sharpe ratio. The strategy found a pattern that was never there. It found faces in clouds.

Complexity is the warning sign. A strategy with 11 tuned parameters that beats the market by 4% a year is almost certainly an overfit. Eleven dials give you eleven chances to bend the past into shape. A strategy with one parameter and an economic reason it should work deserves at least a hearing. An economic reason is a real-world cause, such as a company earning more profit, that explains why the pattern should repeat.

Simplicity is not a matter of taste here. Every added setting is one more chance to fit random movement instead of a real pattern. That is the whole argument for keeping a strategy simple. Count the dials. Then ask what each one earns you.

The Multiple-Comparisons Problem

Testing many strategies guarantees that some will look good by luck alone. This is the multiple-comparisons problem (the more ideas you test, the more false winners you find). It is the deeper issue behind most failed backtests. And it is the one most people miss, because the failures never get published.

Suppose you test 100 random strategies on 20 years of data. At a 5% significance level, you expect about five of them to look statistically significant by pure chance. A 5% significance level means a result that chance alone would produce 5% of the time. Statistically significant means a result that is unlikely to come from chance alone.

Think of 100 people flipping coins. A few of them will flip heads five times in a row. That result looks special. It is only luck, plus a large crowd. The same thing happens when a researcher tests 100 strategies. Chance alone hands over about five apparent winners. Nobody calls those coin flippers skilled. The same doubt should apply to the published strategy.

Then the researcher publishes the best one. You see the winner. You never see the 99 losers that got thrown away. Would you trust a magician who only showed you the tricks that worked? That strategy survived a selection process designed to find lucky accidents that look like patterns. This is why published backtests cluster suspiciously just above the line for significant. The line is not where truth starts. It is where the sorting stops.

The honest correction is strict. Divide your significance threshold by the number of things you tried. That rule is the Bonferroni correction (an adjustment that tightens the bar when you test many ideas). If you tested 100 strategies, your bar becomes 100 times stricter. Few published results survive that math. That fact alone should tell you something.

Better still, demand out-of-sample confirmation (a test on data the strategy never saw during development). A holdout period (a block of recent data set aside and never used while building the strategy) provides that test. It is like grading a student on a new exam, not on the practice test the student already memorized. A test without a holdout period is a rough draft, not a result.

How to Read a Backtest Without Fooling Yourself

Four questions separate a result worth trusting from a result worth ignoring. Ask them every time. No exceptions.

One. How many variations were tried before this one? If the answer is a few, multiply your skepticism accordingly. A strategy picked from 100 tries is far weaker evidence than a strategy tested once. If the author does not say, assume the number was large. Silence on this point is an answer of its own.

Two. Is there an economic reason this should work, or only a shape in the numbers? A statistical pattern is a shape in the numbers with no real-world cause. Patterns without reasons are usually chance wearing a costume. A real reason sounds like this: companies that earn more money tend to be worth more money. A fake reason sounds like this: the line went up on Tuesdays.

Three. What happens on out-of-sample data and in different market regimes? A market regime is a long stretch when markets behave one way. A bull market is a long rise. A bear market is a long fall. A strategy that only works in bull markets is a bull market with extra steps. You want proof it survives more than one kind of weather.

Four. Are transaction costs, slippage, and taxes included? Transaction costs are the fees you pay to trade. Slippage is the gap between the price you expected and the price you actually got. Many beautiful results die the moment you pay to trade them. A strategy that trades often pays often. Paper profits ignore that bill. Real accounts do not.

The base rate is what usually happens. The overwhelming majority of edges found this way decay or vanish. An edge is a real advantage that earns extra return. Most backtested edges are not edges at all. They are accidents that looked like edges until fresh data showed otherwise.

Start from the assumption that a backtest is lying, and make it prove otherwise. That single habit will save you more money than any indicator. Treat every backtest as guilty until proven innocent. The burden of proof sits with the backtest, not with you.