MECHANICAL SYSTEM TRADERS READ THIS CAREFULLY

Your trading systems never performed as expected? It might have been your backtesting program's fault, not your bad luck.

A version of one of the most popular trading programs had a flaw in its backtesting engine and that may have adversely impacted thousands of trading system developers and traders. In this paper I identify the flaw with specific detailed examples and I then compare the results to those obtained from another popular program that offered ways of getting around this type of flaw.

Introduction

Backtesting is the process of evaluating the historical performance of a trading system. Technical traders employ backtesting in an effort to identify historically profitable trading systems. It is a normal practice for most technical traders to discard trading systems that do not perform well when tested on historical data, while at the same time being aware of the fact that past performance is not guarantee of future performance or success.

The integrity of the backtesting process is of great importance to the proper historical evaluation of trading system ideas. Backtesting should produce a realistic and accurate account of the historical performance of a trading system otherwise it may cause rejection of some good ideas and adoption of some bad ones. If the backtesting process is flawed, that could cause a trader to make wrong selections of trading systems and that can lead to devastating losses. A trader that relies on a flawed process to develop trading systems is doomed to fail from start with a high probability. 

I will demonstrate here how a version of one of the most popular, if not the most, backtesting programs that was widely used by thousands of traders in the late 1990s - and it may still being used by many – did not produce accurate backtesting results in many cases due to the way it handled entry and exit signals. As a matter of fact, under certain conditions this program could generate chaotic backtesting results, in the sense that a small change in the initial conditions of a test, such as a slight change of the starting date, could lead to a large variation in calculated trading system performance. I also compare the results to those of another popular program that offered effective ways of getting around this kind of flaws.

It is quite possible that thousands of users of this flawed backtesting program took the integrity of the results for granted just by relaying on the reputation of the vendor and the fact that the software program was widely used. The users possibly assumed that when the backtesting results were not satisfactory, then their idea was bad, and when they were satisfactory, the idea had some merit to it. In both of these cases, if the results were not accurate, the user of the program was the ultimate loser and was cheated, in a way.

This was an excerpt from a 23-page long paper with specific details and backtesting results.

Email at: backtesting@tradingpatterns.com to request a copy of the full paper in pdf format. Please include your name, location and which backtesting programs you have used with the version numbers. If your backtesting program is the same as the program studied in the paper you will be added to a list for receiving the full paper. Warning: emails received from free webmail services (yahoo!, gmail, hotmail, etc.) will not be answered.

A condensed version of the paper with no reference to specific products can be found here.

Study originally performed during August 2009 and revised during August 2010.