Backtesting and Forward Testing
A backtest shows how a strategy would have done on the past; forward testing proves it on live data, and live results are almost always worse than the backtest promised.
OverfittingThe backtest soars while live stalls and starts to slip.
Drag the chart above from Simple to Overfit: the green backtest curve smooths into a flawless line while the red live curve peaks early and then falls apart. That gap has a name, overfitting, and staying on the safe side of it is what this module is really about. Before you risk a single dollar on a rule or a model, you test it, and there are exactly two honest ways to do that plus a whole menu of ways to fool yourself. This module is about telling them apart. A good test can save you from a bad idea. A rigged one, usually rigged by accident, does the opposite: it hands you the confidence to lose money at speed. By the end you will know how to run a backtest that is worth trusting, how to spot the tricks that make a losing strategy look like a money machine, and why the number your test prints is a ceiling you will never quite reach in real life.
Backtesting: replaying a strategy on the past
Backtesting means running a strategy over historical data to estimate how it would have performed. You take your rules, point them at a few years of price history, and let them "trade" that history to see what the equity curve looks like.
It is necessary. A strategy that loses on the past is not worth trading with real money, so a backtest is the floor you have to clear first. But clearing the floor is not the same as reaching the ceiling. A backtest is easy to rig into showing results the strategy could never have earned in real time, and most of the time nobody rigs it on purpose. The mistakes creep in quietly.
So treat a good backtest as permission to keep investigating, never as proof you will make money. The rest of this module is the list of ways a clean-looking backtest lies to you.
Look-ahead bias: cheating with tomorrow's newspaper
Look-ahead bias is using information in your test that was not actually available at the moment of the simulated decision. The test "knows" something the real trader could not have known yet, so it makes a decision no real trader could have made.
Two classic examples:
- Using a day's closing price to trigger a trade you pretend to place "at the open" that same morning. In real life you do not know the close until the day is over.
- Using restated fundamentals that were only published weeks after the date you are testing.
Even a tiny leak of future information turns an ordinary strategy into a fantasy of smooth, effortless gains. The equity curve looks gorgeous because the backtest is quietly reading tomorrow's newspaper. If a backtest looks too clean and too easy, look-ahead bias is the first thing to suspect.
Curve-fitting, and the in-sample / out-of-sample fix
The second great trap is curve-fitting. If you keep tuning a strategy's parameters until its historical curve looks perfect, you have not found a real edge. You have moulded the strategy to the exact bumps of the past, including the random noise that will not repeat. The curve is beautiful precisely because you shaped it to that one stretch of history.
The standard defence is in-sample versus out-of-sample testing, and it is simpler than it sounds:
- Split your history into two slices.
- In-sample: tune the strategy on the first slice as much as you like.
- Freeze the rules. No more tweaking.
- Out-of-sample: run those frozen rules on the second slice, which the strategy has never seen.
If performance holds up out-of-sample, the edge might be real. If it collapses, you were fitting noise, and better to learn that now than with live money. A rule of thumb worth carrying: a strategy with dozens of parameters and a suspiciously smooth curve is guilty until proven innocent.
Forward testing: the honest dress rehearsal
Forward testing, also called paper trading, means running the strategy live on real-time data with no real capital at stake. It is the honest complement to backtesting for one reason: you cannot peek at the future. The data arrives one tick at a time, exactly as it would in real trading, so look-ahead bias and quiet data-snooping have nowhere left to hide.
Forward testing also checks the boring things a backtest glosses over: your order types, your timing, and how fills actually behave when the market is moving. Those execution assumptions are where a lot of "perfect" backtests fall apart.
Think of it as the last checkpoint before real money. Skipping it is how a beautiful backtest becomes an ugly live account.
Why live is always worse than the backtest
Here is the uncomfortable pattern: the gap between backtested and live results is real, it is predictable, and it almost always runs the same direction. Live is worse. Three forces cause it.
| Force | What it is | Why the backtest missed it |
|---|---|---|
| Costs | Fees, bid-ask spreads, funding | A naive backtest often assumes zero costs |
| Slippage | The gap between the price you expected and the price you got | It grows in fast or thin markets a backtest treats as calm |
| Regime change | The market conditions that made your data are gone | History cannot contain a market that has not happened yet |
Because of this, forward-test long enough to see many trades and at least one shift in market conditions before you commit capital. Count trades and range of conditions, not calendar days. A fast strategy might need a few weeks; a slower one often needs months. The question is never "how long has it run?" but "how much has it survived?"
The cost math that quietly kills a backtest
A backtest reports a strategy earning +40% a year on a $10,000 account, with a gorgeous smooth curve, trading 300 times a year. The backtest assumed zero costs. Now add reality.
Say each round-turn trade costs roughly 0.1% of the traded value in fees and spread. Across 300 trades that is about 300 x 0.1% = 30% of turnover eaten by costs. Then add a few basis points of slippage per trade in the fast moments, and it gets worse.
Recomputed with costs, that +40% backtest might realistically land closer to +8% to +10% live. A strategy that looked merely mediocre can flip to a net loss once the costs are honest. Nothing about the logic changed. The only difference was counting the costs the backtest ignored. This is why you subtract realistic costs and slippage before you get excited, and why forward testing (where costs are unavoidable) is the real proving ground.
A cleaner way to split in-sample and out-of-sample
The basic split (tune on slice one, test frozen on slice two) is enough to catch most curve-fitting. If you want to be stricter, keep the out-of-sample slice truly sealed:
- Decide the split before you look at any results, not after.
- Test on the out-of-sample data once. If you keep peeking, adjusting, and re-testing, that "unseen" slice slowly becomes in-sample too, and the whole benefit leaks away.
- Prefer a strategy that stays merely good across both slices over one that is dazzling on one and shaky on the other. Consistency across data it never saw is the signal you actually want.
How long is 'long enough' to forward test?
There is no fixed number of days, because two strategies can trade at wildly different speeds. The honest measure is exposure to variety:
- Enough trades that a lucky streak cannot flatter the record. A dozen trades tell you almost nothing; a few hundred start to mean something.
- At least one change in conditions, so you have seen the strategy in a range and in a trend, in calm and in a scare, not just in the one mood it happened to launch in.
A high-frequency strategy might tick both boxes in weeks. A slow swing strategy might need many months. If your forward test only ever saw one kind of market, you have not really tested it yet.
Quick knowledge check
Why is a backtest necessary but not enough on its own? A strategy must at least survive the past to be worth trading, so a backtest is the floor. But backtests are easily distorted by look-ahead bias, curve-fitting, and ignored costs, so passing one does not prove you will make money live.
What is look-ahead bias, in one example? Using information in the test that was not available at the decision moment, for example triggering a trade "at the open" using that day's closing price, which no real trader would know yet.
Why do live results almost always fall short of the backtest? Three forces the backtest usually understates: costs (fees, spreads, funding), slippage (worse fills than expected), and regime change (the market conditions that produced your historical data have shifted).
Sources
- CFA Institute, "Backtesting and Simulation", definition of testing a strategy on historical data to estimate performance.
- arXiv (academic), "Look-Ahead-Freedom as Temporal Non-Interference", using information not available at the simulated decision time.
- Notices of the AMS (academic), "Pseudo-Mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance", curve-fitting as describing noise instead of the underlying relationship, and failing to generalise to new data.
- CME Group, "About the Trading Simulator", forward testing on real-time data with no real capital before going live.