Counter-Intuitive Facts About Backtesting
by Michael Harris, Price Action Lab
One would think that by backtesting more ideas and more frequently, the chances of discovering an edge increase. However, the opposite exactly happens: the chances of discovering something of value diminish with frequent backtesting.
If you believe this is counter-intuitive you could try to think about it in a different way: If the frequency of backtesting made a difference in discovering an edge, then all those traders who struggle with machine learning algorithms would instead be rich. So what is the process that turns backtesting into a an exercise in futility as time goes by?
The answer is data-mining bias. As time goes by and a trading system developer tests more ideas, new or modified, data-mining bias increases and at some point it becomes so large that the probability of discovering an edge via backtesting gets close to 1. This essentially means that there is a two-way interaction between the one who is backtesting and the backtesting process. This is the crucial element missing when traders believe that it is counter-intuitive that the frequent use of backtesting decreases the chances of success. This is illustrated in the figure below:
In the above figure, “a” is in general a non-liner function of N and of other parameters that assumes a value of less or equal to 1. This means that the probability P tends to 0 as N becomes large. Essentially, those that backtest constantly have little chances of finding an edge. This includes machine learning programs that mine data relentlessly until something is over-fitted in the in-sample and curve-fitted in the out-of-sample by luck (See this article for more details.)
The conclusion is that traders should use backtesting only when there is a good idea to test. Actually, the idea must be as unique as possible. Then, the objective should be to try to debunk it, not prove that it is good by adding more filters and conditions. One of the fundamental problems of users of backtesting is that they try to prove that an idea is good when they should be trying exactly the opposite. The reason for this, which is also a counter-intuitive notion, is that there are very few edges and the probability that any one idea you backtest is a fluke is close to 1 to start with.
These and other facts about backtesting that are rarely or never discussed, along with specific rules and examples of how to deal with data-mining bias, can be found in my newly released book below:
Publisher: Michael Harris
Date: September 1, 2015
Language: English
270+ pages (6″ x 9″ trim)
74 high quality charts
Available online only
Table of Contents
You can subscribe here to notifications of new posts by email.
Copyright © Price Action Lab
