Why backtests are useless, EAs are flawed and their parameters are bad [DISCUSS]

[Tyger]

If you can describe a genetic algorithm using a primitive spreadsheet modelling tool, be my guest. until then what Shakone is describing is hardly snakeoil and certainly not tosh

1). I was genuinely curious about what an "adaptive" strategy might be. When I hear things like "...it can't be easily categorised, figured out or copied ..." then my scientific brain sets off alarm bells. If we can't categorise, figure out or copy it then how on earth can we test it? And if we can't test it, how do we know it's real? We start getting into the realms of faith. I prefer evidence based facts - everything else goes into the snakeoil bin. The scientific revolution was called the Age of Enlightenment for good reason.
2). Google Genetic Algorithm Excel - there are numerous examples available.
3). Excel a "primitive spreadsheet modelling tool"? I find it an advanced high-level programming language. Different strokes for different folks I guess.

Back to the discussion: If anyone can give me a clear and concise example of an "adaptive" strategy, or system, then I'd be really keen to hear it. If I google "adaptive trading system (or strategy)" then all I get is the usual snakeoil websites. I'm a scientist first, and a quant second. I'm curious. I'm keen to find out the facts. I suspect that the word "adaptive" in trading is snakeoil. BUT I MIGHT BE WRONG. If anyone can help, then I'm all ears.

 

[Triggerfish]

"adaptive" to what may I ask?
Surely one cannot have adaptive without something else?.....so what is it?

 

[malaguti]

1). I was genuinely curious about what an "adaptive" strategy might be. When I hear things like "...it can't be easily categorised, figured out or copied ..." then my scientific brain sets off alarm bells. If we can't categorise, figure out or copy it then how on earth can we test it? And if we can't test it, how do we know it's real? We start getting into the realms of faith. I prefer evidence based facts - everything else goes into the snakeoil bin. The scientific revolution was called the Age of Enlightenment for good reason.
2). Google Genetic Algorithm Excel - there are numerous examples available.
3). Excel a "primitive spreadsheet modelling tool"? I find it an advanced high-level programming language. Different strokes for different folks I guess.

Back to the discussion: If anyone can give me a clear and concise example of an "adaptive" strategy, or system, then I'd be really keen to hear it. If I google "adaptive trading system (or strategy)" then all I get is the usual snakeoil websites. I'm a scientist first, and a quant second. I'm curious. I'm keen to find out the facts. I suspect that the word "adaptive" in trading is snakeoil. BUT I MIGHT BE WRONG. If anyone can help, then I'm all ears.

Hi Tyger, an example of an adaptive strategy, is just what I've mentioned..a genetic algorithm. These can be back and forward tested, however I'm afraid they dont fit nicely into an excel formula. You need to program these things, so when you google excel and genetic algorithm you now get into the realms of programming. Add ons to excel that utilise the very primitive language of visual basic.
So we have exactly what Shakone has described and something that can't fit into your excel formula (or else we wouldn't need add ons and the use of a programming language). So, still Tosh?
Thats the only point I'm trying to make.

 

[Darwin-FX]

Back to the discussion: If anyone can give me a clear and concise example of an "adaptive" strategy, or system, then I'd be really keen to hear it. If I google "adaptive trading system (or strategy)" then all I get is the usual snakeoil websites. I'm a scientist first, and a quant second. I'm curious. I'm keen to find out the facts. I suspect that the word "adaptive" in trading is snakeoil. BUT I MIGHT BE WRONG. If anyone can help, then I'm all ears.

Well I thought this is clear, but let me try to explain it.

The underlying problem with the markets is that the data we process changes all the time.
So a relationship that existed in the past, eg "When the price dropped 50 pips in 3 minutes, then a counter-movement will happen 60% of the times" (completly made up, just for an example) will no always be the same!

For example a month later it could be "When the price dropped 40 pips in 2 minutes, then a counter-movement will happen 55% of the times".
Tough the underlying inefficiency we want to exploit (counter-movements after price movements) stays the same, the exact parameters to describe it correctly will change.

So we need a way to always "get the best parameters" for the current market (not for the market 5 years ago).

And to always get the best parameters for a strategy, to match the current manifestation of the inefficiency we want to exploit, we need to adapt, which is nothing more than to always re-optimise the parameters we use when time goes by.

This is all my current tool can handle, tough, I am working on methods that will even be capable of replacing the whole trading logic, not only the parameters, based on what is working best in the current market situation.

Hope that makes it clear - if not ask me, thats what the thread is about.

-Darwin

 

[ethanslad]

nice thread guys...

 

[Shakone]

...

1). I was genuinely curious about what an "adaptive" strategy might be. When I hear things like "...it can't be easily categorised, figured out or copied ..." then my scientific brain sets off alarm bells. If we can't categorise, figure out or copy it then how on earth can we test it? And if we can't test it, how do we know it's real? We start getting into the realms of faith. I prefer evidence based facts - everything else goes into the snakeoil bin. The scientific revolution was called the Age of Enlightenment for good reason.
2). Google Genetic Algorithm Excel - there are numerous examples available.
3). Excel a "primitive spreadsheet modelling tool"? I find it an advanced high-level programming language. Different strokes for different folks I guess.

Back to the discussion: If anyone can give me a clear and concise example of an "adaptive" strategy, or system, then I'd be really keen to hear it. If I google "adaptive trading system (or strategy)" then all I get is the usual snakeoil websites. I'm a scientist first, and a quant second. I'm curious. I'm keen to find out the facts. I suspect that the word "adaptive" in trading is snakeoil. BUT I MIGHT BE WRONG. If anyone can help, then I'm all ears.

Curious about something that is tosh?

You make a declaration, that fixed strategies can't be successful for long (declaring your belief). I ask you whether adaptive strategies have the same flaw. You then ask for an example (requesting info) of an adaptive strategy but suggest that it is snakeoil (stating your bias). I answer, but don't give you the answer you want (feedback). Because the answer doesn't satisfy you, you then say that you think it's a load of tosh (dismissive), but ask me to feel free to prove you wrong (manipulative). Now you're all ears (requesting info again) and admit you might be wrong (humble).

So from declarations of certainty, to requesting info, to stating bias, to being dismissive, to being manipulative and back to being curious and open to being wrong.

And you think that there is no such thing as an adaptive strategy?

I don't believe that the direction and timing of market movements will fit easily into a formula in excel for you, but that's just my belief and shouldn't stop you trying. I encourage testing everything you can. Lots of things are backtestable, some things aren't. Does that automatically mean those that aren't are bad? I wouldn't go as far as the OP to say backtesting is useless, but forward testing is more useful to me.

 

[Darwin-FX]

I wouldn't go as far as the OP to say backtesting is useless, but forward testing is more useful to me.

That was just a statement to start the discussion, I know that a backtest can be usefull if done right.

But doing it right is soooo much harder than loading an EA and click "Backtest" in metatrader, and this is something lot traders (especially new ones) do not understand.

In opposite, its much harder to get useless informations out of a walk forward analysis.

-Darwin

 

[numbertea]

I could be wrong, but aren't all back testers who trade their back tested systems and then study their trading results using walk forward analysis? That's what I do. After I feel that I have enough data to backtest and I build my system to work as I can on that data, I forward test on out of sample data going forward... This is called trading, also called walk forward analysis now I guess. Cool.

Cheers

 

[Darwin-FX]

Sure, thats the purpose, simulate what a trader does when he learns and adapts strategies.. but the difference is that an algorithm can do everything automated, taking you 1 minute to set it up, and not hours of hard work

Also, before you go live with a system, the algo can test the used adaption-methodology on the past decade and tell you if it is really worth trading, something you won't be able to do in an efficient way when you do it manually.

But, on the other hand, a good trader might be better in adapting.. but I'm working on that part

-Darwin

 

[malaguti]

I could be wrong, but aren't all back testers who trade their back tested systems and then study their trading results using walk forward analysis? That's what I do. After I feel that I have enough data to backtest and I build my system to work as I can on that data, I forward test on out of sample data going forward... This is called trading, also called walk forward analysis now I guess. Cool.

Cheers

not necessarily Numbertea, I could back test my system up to 2011 and then forward test on out of sample data, which would be 2012 and 2013 say. If it still stacks up, then I'm good to test on demo. Thats how I prefer to back test which is why I disagree completely with the original thread.
Often a backtest will include data right up to the last day, which in my opinion is a slightly flawed approach
I walk forward test on half decent backtested results. If those results were no good, I would never waste my time forward testing something. Back testing results is crucial.
Thats just my opinion of course, and doesn't need a thread to discuss it.

 

[Darwin-FX]

not necessarily Numbertea, I could back test my system up to 2011 and then forward test on out of sample data, which would be 2012 and 2013 say. If it still stacks up, then I'm good to test on demo. Thats how I prefer to back test which is why I disagree completely with the original thread.

What you do there is out-of-sample testing, which means you have 1 dataset to optimise your strategy and then 1 dataset to test your optimisation approach.

This is ok, better than a backtest at least.

But are you aware of the fact that you judge the future performance of your strategy based on only 1 datapoint? (in-sample as "past data", out-of-sample as "future data" => 1 datapoint to judge live trading)

A good Walk Forward Analysis can give you at least ~100-150 datapoints to judge from. The approach stays the same, but the results are more reliable as you view a larger part of the picture

So even if you disagree with my post, its a matter of fact that a WalkForwardAnalysis is just the way you do it - but one step further

Perhaps this convinces you to give it a try, at least

-Darwin

 

[Tyger]

Hi Tyger, an example of an adaptive strategy, is just what I've mentioned..a genetic algorithm. These can be back and forward tested, however I'm afraid they dont fit nicely into an excel formula. You need to program these things, so when you google excel and genetic algorithm you now get into the realms of programming. Add ons to excel that utilise the very primitive language of visual basic.
So we have exactly what Shakone has described and something that can't fit into your excel formula (or else we wouldn't need add ons and the use of a programming language). So, still Tosh?
Thats the only point I'm trying to make.

Mea Culpa
I was wrong. I should have said formula (or algorithm) instead of Excel formula. What I'm trying to say is:
The scientific approach: Evidence based analysis using testable methods and objective evidence, rigorously and skeptically examined.
Any other approach: Theories built on untestable propositions, anecdotal evidence and intuitive analysis.
Tell me more about GAs (or EAs). Are they not just enhanced versions of Excel's solver function? In other words, sophisticated data-mining tools?

 

[Tyger]

Curious about something that is tosh?

You make a declaration, that fixed strategies can't be successful for long (declaring your belief). I ask you whether adaptive strategies have the same flaw. You then ask for an example (requesting info) of an adaptive strategy but suggest that it is snakeoil (stating your bias). I answer, but don't give you the answer you want (feedback). Because the answer doesn't satisfy you, you then say that you think it's a load of tosh (dismissive), but ask me to feel free to prove you wrong (manipulative). Now you're all ears (requesting info again) and admit you might be wrong (humble).

So from declarations of certainty, to requesting info, to stating bias, to being dismissive, to being manipulative and back to being curious and open to being wrong.

And you think that there is no such thing as an adaptive strategy?

I don't believe that the direction and timing of market movements will fit easily into a formula in excel for you, but that's just my belief and shouldn't stop you trying. I encourage testing everything you can. Lots of things are backtestable, some things aren't. Does that automatically mean those that aren't are bad? I wouldn't go as far as the OP to say backtesting is useless, but forward testing is more useful to me.

I'm envious of your social skills. I don't understand most of what you said on that topic, but that's typical of people on the autistic spectrum. If I offended you then I apologise unreservedly.

 

[Shakone]

I'm envious of your social skills. I don't understand most of what you said on that topic, but that's typical of people on the autistic spectrum. If I offended you then I apologise unreservedly.

Not offended at all, and no need to apologise.

So would you concede that you apply adaptive strategies to all kinds of situations in your life, that aren't statistically verified on historical data, or is it still something that doesn't exist? Looks like I've failed in my attempt.

 

[trendie]

I am still curious about how a walk-forward is any different to testing on out-of-sample data, and how being adaptive is anything other than having just another layer of test clauses to trigger a trade.

I know I am being thick, but still would like to see an example to illustrate.

 

[numbertea]

I am still curious about how a walk-forward is any different to testing on out-of-sample data, and how being adaptive is anything other than having just another layer of test clauses to trigger a trade.

I know I am being thick, but still would like to see an example to illustrate.

I am not sure if this would satisfy what you are asking for in an 'adaptive strategy' but I use the market volatility before 9:30am to give my system a volatility level that then will affect which algorithms will be used during my trading hours of that day. Is that a good example? I of course realize that it is just a bunch of more test clauses. And like you I don't know the increased significance of calling trading with no money "walk forward analysis". Can someone turn on the light?

Cheers

 

[Tyger]

Well I thought this is clear, but let me try to explain it.

The underlying problem with the markets is that the data we process changes all the time.
So a relationship that existed in the past, eg "When the price dropped 50 pips in 3 minutes, then a counter-movement will happen 60% of the times" (completly made up, just for an example) will no always be the same!

For example a month later it could be "When the price dropped 40 pips in 2 minutes, then a counter-movement will happen 55% of the times".
Tough the underlying inefficiency we want to exploit (counter-movements after price movements) stays the same, the exact parameters to describe it correctly will change.

So we need a way to always "get the best parameters" for the current market (not for the market 5 years ago).

And to always get the best parameters for a strategy, to match the current manifestation of the inefficiency we want to exploit, we need to adapt, which is nothing more than to always re-optimise the parameters we use when time goes by.

This is all my current tool can handle, tough, I am working on methods that will even be capable of replacing the whole trading logic, not only the parameters, based on what is working best in the current market situation.

Hope that makes it clear - if not ask me, thats what the thread is about.

-Darwin

Excellent response. But can I play Devil’s Advocate? (please don’t take offense if I appear confrontational – I don’t mean to be!).

So “adaptive” means “changing parameters to optimise results as time goes by”?

So you come up with a theory that a “relationship” truly existed in the past (and was not just data fitting). You surmise that the relationship continues into the future but that the parameters that describe it change (for whatever reason).

So do you then

a). try and find a new relationship that describes how the original parameters change with time? (eg, using your hypothetical figures, we might find that the market cycles between your two examples every month, or we might find a direct relationship to the S&P ATR (average true range), for instance).

Or

b). are you speculating that “if the optimum parameters for the current market are A, B and C, if we trade using these parameters then we can expect X% return”? In this instance, we would, for example, go back 3 years then walk forward each month, “optimising” as we go, then seeing how well this approach worked for the next month, etc, etc. Then if we get sufficiently good results over, say, the last 3 years, we confidently conclude that we have a viable trading strategy.


But surely a). above would be a “Holy Grail” of trading strategies (thus extremely unlikely), and b). is just data mining/fitting? (ie I could do the same with a series of random coin tosses and it would make money most of the time, but be subject to occasional blow-outs).

PS Great thread. Enjoying it immensely!

 

[malaguti]

What you do there is out-of-sample testing, which means you have 1 dataset to optimise your strategy and then 1 dataset to test your optimisation approach.

This is ok, better than a backtest at least.

But are you aware of the fact that you judge the future performance of your strategy based on only 1 datapoint? (in-sample as "past data", out-of-sample as "future data" => 1 datapoint to judge live trading)

A good Walk Forward Analysis can give you at least ~100-150 datapoints to judge from. The approach stays the same, but the results are more reliable as you view a larger part of the picture

So even if you disagree with my post, its a matter of fact that a WalkForwardAnalysis is just the way you do it - but one step further

Perhaps this convinces you to give it a try, at least

-Darwin

You misunderstand me, I concur with the whole approach of forward testing, I don't need to be convinced. But when someone says backtesting is useless, its so far off the mark its just not worth discussing.
I have generalised my forward testing approach just for ease of understanding, I may indeed have multiple data points..but the real forward testing is doing it live, on demo. None of the tests will account for slippage, late entries, gaps etc etc.

 

[new_trader]

You misunderstand me, I concur with the whole approach of forward testing, I don't need to be convinced. But when someone says backtesting is useless, its so far off the mark its just not worth discussing.
I have generalised my forward testing approach just for ease of understanding, I may indeed have multiple data points..but the real forward testing is doing it live, on demo. None of the tests will account for slippage, late entries, gaps etc etc.

100% correct and the very reason why live trading results vary widely from automated back testing.

IMO: The only way to learn trading is to practice day and night and weekends using a simulator and historical data in real time. Practice, practice study and practice.

 

[Shakone]

Harris Brumfield:

You need to have some kind of game plan. But things evolve so quickly in the market you have to be able to adapt. Combining some kind of systematic foundation with the ability to change things on the fly probably gives you the best of both worlds.

I would read a lot before the trading day, and even during the day, to find a theme, have a direction and develop a trading bias. No market pattern repeats exactly the same, but you can see a lot of things developing [like they did in] the past if your memory is good enough and you remember things that have happened repeatedly.

But you have to mix things up and adjust for the time. One of the problems I saw is that technical ideas eventually get caught and you have to be able to shift real fast. Approaching things the way I did, I figured if I ever found anything [that was really good], it would be harder for people to discover and copy.