Why backtests are useless, EAs are flawed and their parameters are bad [DISCUSS]

[Darwin-FX]

Hi, I am Darwin and I am new here.

tl;dr In the next weeks, I am planing to release my private Walk Forward Analyzer, but before I do this, first I want to make sure that also the beginners in mechanical trading understand why common backtests are quite useless and why walk forward analysis is better in so many ways, so I decided to start this discussion.

I WILL RELEASE THIS TOOL FOR FREE AND OPEN SOURCE!

And as people keep asking why I release free stuff, let me explain my motivations: I am targeting a job in the trading-economy, and this is not easy to get. Beeing known in the community would help me a lot to accomplish this goal, so I decided to release some of my private tools for free and/or open source to make myself a name over the next months. I am not here to sell you stuff, keep that in mind, please!


For the record: I do not claim that WFA is the holy grail in system testing, but I argue that it is way better than normal backtests.




THE GENERAL APPROACH TO SYSTEMATIC TRADING

Take a trading logic => Generate an EA that trades exactly like it => Optimise Parameters => Do a backtest => Trade it live => $$$

BUT: Everybody that has ever had an EA that was a money-printing machine in the backtests (and this is very, very easy to be done), knows that a good backtest does not imply profitable live trading.
And that is the problem. We need a test-method on whichs evaluation we can rely!




STATEMENT 1: Your parameter choices are not good

Well, every EA consists of 2 things:

1. the logic/script itself
2. the parameters (like periods for moving averages, stop loss values etc. Just everything that can be adjusted!)

The first thing is static and "given". And a lot traders only focus on this part.
But, an EA can behave in very different ways, depending on the parameters. And there can be billions of possible parameter-combinations (=parameterset). There are 3 approaches to determine the parameterset for live trading.

1. the intuitive approach (non adapting):
The trader just chooses the parameters based on his expert knowledge.
But, there are just too many possible parametersets. You can't just "guess" them without evaluation and testing.
It does not matter how good you are, you are never good enough to "think through" this huge amount of possibilities in a reliable fashion!

2. the "optimise on all data" approach (non adapting):
The trader chooses the parameters based on all past data.
So, the configuration/parameters that worked best for the last 12 years, for example, are chosen to be traded live.
But, to stay up-to-date it is not a very good method to optimise the parameters on so much and therefore OLD data.
Because the market today is not the same as it was 12 years ago.

3. the "optimise on the last few years" approach:
The trader chooses the parameters based on the last few years of data.
But you can not just take this approach without testing how this "optimisation method" would have worked in the past.
And this is exactly what a walk forward analysis will do: "Optimise on the last few years", but it tests this approach on all data in the past!




STATEMENT 2: A non-adapting EA can never make longterm profits

I mean, there is an infinite amount of possible trading-systems AND an infinite amount of ways the market can change AND an "infinite" amount of possible parameters for your trading-system.

Considering this, would you really want to bet money on the fact that you have a trading system that will always work, in the same way, without adaption, on all future market conditions? I would not.

"But why shouldn't I be able to put a traders knowledge into a script and trade it?"
Its simple: A trader always learns, he takes input from many sources, he has knowledge about the markets and therefore he adapts his trading strategy so he can always be as close to the markets as possible.
A simple EA script can't do this, a Walk Forward Analysis can (to some extent).

Reason #1 why there are many "profitable" EAs out there
The EA really works, it has a sound strategy and was developed properly. But the markets always change, and they can change in infinite ways.
So, at some point, the traded market-inefficiency WILL(!) change, and the EA can not adapt to this change and therefore will lose its profitability.

Reason #2 why there are many "profitable" EAs out there
is that some of them, sound or unsound systems, are just lucky.
If you send 10.000 people to a casino and let them play for a while, just due to chance, some of them will make profit over some time.
The same is for EAs, if there are enough of them, some will really make profit (even if they are in fact useless).
But as they keep trading, they will lose, as the probabilities are against them (same as in a casino).

Reason #3 why there are many "profitable" EAs out there
is the huge risk some EAs take (grid trading, martingale systems etc).
These EAs take very huge risks to make small profits. But somewhen, the risk strikes, and at that point, the EA will to lose (all) money.

Reason #4 why there are many "profitable" EAs out there
is the small timespan on which they are profitable. Its no magic to make profit for months, but its hard to make it for many, many years.




STATEMENT 3: A backtest does not tell you anything about the future performance

Well, don't get me wrong, I strongly hold the opinion that simulations on the past are the only way to really test a trading system.
But what does a backtest tell you? Just that your system performed well on the past.

But is trading really about having a system that represents the past?
No, trading is about designing a system on the past and then trade it in the future.

THIS IS A FUNDAMENTAL DIFFERENT QUESTION THAN WHAT A BACKTEST CAN TELL YOU!
You want to answer the question "how good is my live trading performance" with the answer to the question "how good did my system perform in the past". That logic is flawed, of course.

"But if I use out-of-sample data to verify my backtest..."
Out of sample testing is a good idea. But you only have one optimisation-dataset and one test-dataset, which is not very reliable. Walk forward analysis, somehow, is out-of-sample testing on steroids. It uses the same method, but generates 10-1000 opti/test dataset-pairs.

"So, are all backtests useless?"
No, it can help you to get sound and good trading systems, if done right.
But my point is that a good performance in a backtest does not make sure the system is also good in live trading.

Trading is a game of probabilities, and the probability that an EA is profitable, only based on a good backtest, is very, very low.

"But why should a system that performed good over the past simply stop working"
The main reason is, and that might be the most important one, OVERFITTING.
That means, the EA tackles an "random" inefficiency of the market that was just there in the past, but that was no real thing.

I found a very good picture that explains overfitting:

You see? There are infinite of such useless "relationships" in the price data, and most EAs tackle one of them which gives them very good backtest results, but that was just due to luck, and these "relationships" will not hold in the past.
And as there are a lot more of these overfitted solutions than sound ones, the chances are very high that you optimised towards such an overfitted system.

The other reason is that the markets always change. So even if you have a non-overfitted, good performing backtest, you will run into serveral problems that you can read in Statement 2



THE SOLUTION: WALK FORWARD ANALYSIS

A Walk Forward Analyser is an external program that can take EVERY EXPERT ADVISOR and, using Metatrader4, do the analysis.
SO YES; YOU CAN CONTINUE TO USE YOUR ALREADY WRITTEN EAs!

Well, some of you will already know how a walk forward analysis works. For the others here is a short overview, as I will explain the process in more details in a future thread.

A WFA takes, for example, the data from 2000 to optimise your system, then tests it on 2001 (which is, from the point of optimisation, the "future" or "live trading").
Then it walks forward and optimises the system on 2001, test it on 2002. Then optimise on 2002, test on 2003 etc.
Do this until you walked through your whole data and only consider the "live trading" in your evaluation.

You see? With this simple tactic you can tackle the 3 problems I have described above, the lack of adaptability, the need for a evaluated parameter selection process and the uselessness of a "past-performance"-backtest.


And then, when you want to trade the system live, you do the exact same thing that you have tested:
1) Optimise on the last available data, with the same optimisation settings you tested in WFA
2) Trade it in the future, with the same procedure you tested in WFA

And that is the main difference to normal backtests: YOU TRADE YOUR EA IN THE EXACT SAME WAY AS YOU TESTED IT!!!



So, this is what I wanted to say, now feel free to tell me what you think about this topic, please
- Darwin

 

[numbertea]

If the stars align to show a consistent trading method over 100 trades I would trade it. Relationships exist in nature and everything on earth is part of nature. Backtest to find relationships that are timeless and find your holy grail. It is different for each of us. We all observe things differently. I wouldn't discount a relationship but linked ones will usual survive the test of time better. Having faith in your own powers of observation will allow you to trade your back tested theories.

Cheers

 

[NVP]

to me all I do is stress test systems in tough market conditions....any EA or dumbass system performs well in strong market conditions

N

 

[Tyger]

Some very good points, but...

The market is, in mathematical terms, a Complex Adaptive System (CAS) (see Wiki) which means that if anyone starts making excessive profits then the market will adapt. In a nutshell, if you find that buying IBM at the open consistently makes profit, then before long the people from whom you bought the shares will stop selling them to you because they consistently make a loss.

Thus irrespective of how you arrived at your successful trading strategy it won't stay successful for very long.

Constructive criticism of this issue most welcome!

 

[numbertea]

Some very good points, but...

The market is, in mathematical terms, a Complex Adaptive System (CAS) (see Wiki) which means that if anyone starts making excessive profits then the market will adapt. In a nutshell, if you find that buying IBM at the open consistently makes profit, then before long the people from whom you bought the shares will stop selling them to you because they consistently make a loss.

Thus irrespective of how you arrived at your successful trading strategy it won't stay successful for very long.

Constructive criticism of this issue most welcome!

Fear and greed haven't changed. Participants must still buy and sell.

Cheers

 

[Tyger]

Fear and greed haven't changed. Participants must still buy and sell.

Cheers

Very good point. But I estimate that, as of Sept 2013, Algorithmic trading (see Wiki article) comprises around 75 - 80% of all trades. Computers aren't affected by emotions, thus the percentage that market moves are influenced by Fear and Greed is relatively small, and shrinking.

(Further constructive criticism of my opinions is most welcome!!!)

 

[Shakone]

You say a fixed strategy won't be successful for long because it's a complex adaptive system. What about an adaptive strategy?

 

[random12345]

Very good point. But I estimate that, as of Sept 2013, Algorithmic trading (see Wiki article) comprises around 75 - 80% of all trades. Computers aren't affected by emotions, thus the percentage that market moves are influenced by Fear and Greed is relatively small, and shrinking.

(Further constructive criticism of my opinions is most welcome!!!)

The problem with figuring out how much trade is actually algo generated is that algos are used to clip client orders or even large discretionary orders by institutions and these get lumped into the algo pile too, even though they do not originate through what you would call a trading algo, but they hit OTC markets or exchange markets like lightning because of how they are controlled. For example:

http://www.autobahnfx.db.com/algo_slicer_popup.html

Some trades are also put through price improvement strats or risk management systems like Athena and Autobahn and again, are they really algo trades?

Even so you're right, lots of algo trading out there, but so long as the market moves away from the mean and does so for longer than microseconds, then is not yet efficient. There have always been more advanced systems battling one another for supremacy, and less advanced people still doing okay for themselves. Fear and greed apply to some markets more than others, and even so they still manifest in algos - except they're called return on account/dd management and profit factors.

You only need to check the IB/HF job pages to know that algos are here to stay and it's best to find ways to win without thinking too much about your opponent as the latter is wasted energy.

 

[new_trader]

In a nutshell, if you find that buying IBM at the open consistently makes profit, then before long the people from whom you bought the shares will stop selling them to you because they consistently make a loss.

To realise a profit you need to do two things Buy AND Sell. Your nutshell summary assumes that: 1) You are constantly buying from the same person and 2) that person is always selling at a loss because they are always buying at a higher price. It is unrealistic.

Before trading I spray my computer with Algo-Be-Gone...works every time.

 

[Darwin-FX]

Good to see some discussion here

If the stars align to show a consistent trading method over 100 trades I would trade it. Relationships exist in nature and everything on earth is part of nature. Backtest to find relationships that are timeless and find your holy grail

Well, first of all you are right, you can find relationships on your own and then backtest them to see how good they were, and this might work or not.
But nevertheless, also a good backtest might end in a 100% useless system, but a good walk-forward-analysis has a much lower probability on beeing misleading/useless, and thats what it is about


-------------------------------------------------------------------------------------------------------------------------------------

to me all I do is stress test systems in tough market conditions....any EA or dumbass system performs well in strong market conditions

True that. Also, any EAs can perform well in short market conditions using unsound tactics and high risk/reward. So it is never about short-term profits, these are easy to get, it is about the long run, and as you said, the tough market conditions are the interesting ones

-------------------------------------------------------------------------------------------------------------------------------------

The market is, in mathematical terms, a Complex Adaptive System (CAS) (see Wiki) which means that if anyone starts making excessive profits then the market will adapt.
(...)
Thus irrespective of how you arrived at your successful trading strategy it won't stay successful for very long.

Yeah, I think you are right there
As a matter of adaption and change, even the best relationship wont hold, and if it is just because the trends get shorter, or the magnitutes larger, or the retracements weaker.. you name it.

The underlying inefficiency might still remain, but the exact manifestation might change very very fast, which means that the indicators you used to describe an inefficiency might need new parameters very quickly.

-------------------------------------------------------------------------------------------------------------------------------------

computers aren't affected by emotions, thus the percentage that market moves are influenced by Fear and Greed is relatively small, and shrinking.

The problem is that a computer just executes an solution. And the coder determines
a) how the solution works
b) how greedy/risky an algo is
c) when it is allowed to trade/when its execution is stopped

So the emotions stay, tough they are not longer about how to enter/exit trades in particular, but about how and when to execute the algo.

The good thing, with a computer, we can define and evaluate the consequences of a,b and c, which is quite cool

-------------------------------------------------------------------------------------------------------------------------------------

You say a fixed strategy won't be successful for long because it's a complex adaptive system. What about an adaptive strategy?

Thats what walk forward analysis is all about, make every EA an more or less adaptive strategy

-------------------------------------------------------------------------------------------------------------------------------------

Before trading I spray my computer with Algo-Be-Gone...works every time.

:cheesy:

-Darwin

 

[wackypete2]

Before trading I spray my computer with Algo-Be-Gone...works every time.

Great line!

Peter

 

[Tyger]

You say a fixed strategy won't be successful for long because it's a complex adaptive system. What about an adaptive strategy?

Interesting discussion. Can you give us an example of what you mean by an adaptive strategy? It's a term I hear used a lot by "snake oil" salesmen/websites, but I've yet to find anyone who could give me a convincing scientific, evidence-based example.

 

[NVP]

my god .......an intelligent, imformative and well argued thread

what went wrong ?
N

 

[Shakone]

Interesting discussion. Can you give us an example of what you mean by an adaptive strategy? It's a term I hear used a lot by "snake oil" salesmen/websites, but I've yet to find anyone who could give me a convincing scientific, evidence-based example.

An adaptive strategy would be one that has some underlying principles, but doesn't repeat in exactly the same way (so doesn't necessarily have a rigid set of rules for entry and exit that you can predict ahead of time), yet the conditions for entry and exit are still recognised and acted upon by the trader. Because it doesn't need to repeat in the same way and is not rigid, it can't be easily categorised, figured out or copied to the extent that it would become unsuccessful. It might even seem random to someone who doesn't understand the principles.

You probably use adaptive strategies in your life, I'd guess most people do, they just don't care or realise that they are doing it, because it's natural. Think about some things you do in your life, what your goal is, and what your strategy is. Is it always a fixed set of instructions you follow to get what you want or do you adapt your strategy to the conditions as they present themselves to you?

 

[trendie]

This thread seems like a goer.

I always thought the only thing that could be adaptive would be volatility.
Given any entry point, the volatility, or recent range, would determine a variable stop-loss or take-profit.

I also thought being adaptive would be adding "if" clauses to the overall start.
There might be circumstances where the underlying strategy is triggered or not based on additional parameters, such as "not before big news", etc.

I think Shakone has it about right at the moment.

Looking forward to an example of a walk-forward example.

 

[numbertea]

My system sets some variable numbers for the day based on pre market volatility of price and volume per bar.

Cheers

 

[Tyger]

An adaptive strategy would be one that has some underlying principles, but doesn't repeat in exactly the same way (so doesn't necessarily have a rigid set of rules for entry and exit that you can predict ahead of time), yet the conditions for entry and exit are still recognised and acted upon by the trader. Because it doesn't need to repeat in the same way and is not rigid, it can't be easily categorised, figured out or copied .....

I don't mean to be rude or deliberately contentious, but that's a load of tosh. As far as I'm concerned if it can't be described as an Excel formula then it's snake oil. But please feel free to prove me wrong! (It took a long time, but learning to embrace being wrong was the best thing that trading taught me).

 

[Shakone]

An adaptive strategy would be one that has some underlying principles, but doesn't repeat in exactly the same way (so doesn't necessarily have a rigid set of rules for entry and exit that you can predict ahead of time), yet the conditions for entry and exit are still recognised and acted upon by the trader. Because it doesn't need to repeat in the same way and is not rigid, it can't be easily categorised, figured out or copied .....

I don't mean to be rude or deliberately contentious, but that's a load of tosh. As far as I'm concerned if it can't be described as an Excel formula then it's snake oil. But please feel free to prove me wrong! (It took a long time, but learning to embrace being wrong was the best thing that trading taught me).

Ok, as far as you are concerned, it is a load of tosh. I'm fine with that, and you are too.

 

[malaguti]

I don't mean to be rude or deliberately contentious, but that's a load of tosh. As far as I'm concerned if it can't be described as an Excel formula then it's snake oil. But please feel free to prove me wrong! (It took a long time, but learning to embrace being wrong was the best thing that trading taught me).

If you can describe a genetic algorithm using a primitive spreadsheet modelling tool, be my guest. until then what Shakone is describing is hardly snakeoil and certainly not tosh

 

[numbertea]

In C++ the trade determination part of my system is currently 4216 lines double spaced. If I wanted to put all of it in excel and have it working it would take at least 4 times as much code and run crazy slow. I think to be able to describe ideas in plain English for starters is the most important step and then whatever language you want to program it in can easily follow.

Clear, easily described thoughts,first.

I don't think any one if us is extremely inclined to give up algorithms or formulas complete if we are currently basing our livelihood on them. Or is this just me?

Cheers