The elphant in the room, over fitting

[alphadude]

expert system developers,

one of the biggest challenges of building mechanical systems; is whether the system you built has an edge; or simply is curve fitted.

some tools i am using personally are:
- out of sample test
- bootstrap statistics
- forward test (aka paper trading)

what are your thoughts and experiences?

 

[NVP]

Hey Alpha

anyone who has worked with me knows I am not the worlds most proficient MT4 programmer and certainly not the worlds most patient EA developer/collaborator

For me I like to focus on testing systems in that area where they are by definition most weak, ineffectual and exposed to losses

so for a trend based system I will focus on "chop" and flattish market periods and see how it performs during that time in terms of signals/drawdown/losses

Once you have a system that has your "back" (defensive mode) then the rest should take care of itself

for me its defense defense defense before anything else :smart:

N

 

[the hare]

I'm almost tempted to post a sensible reply !

 

[andrewdb]

some of the best comment on this area i have read is in 'way of the turtle'.
plain english and not much math - that may or may not be a reccomendation.

 

[trendie]

ALL systems are essentially curve-fitted.

a: You make a general observation about the market.
b: You formalise the observation into a set of rules.
c: You try out those rules, and test using differing parameters giving optimal returns.
d: You re-test against different data, or forward test.

I concur with andrewdbs recommendation of the Way of the Turtle.
The book has a chapter on optimisation.

System designers should spend more time on robustness.
That is, if a system creates a radically different return by changing parameters just slightly, be very wary, it is most likely curve-fitted.

A good system should generate similar returns even if you mess about with the given parameters, within reason.

 

[Mike Kshemaraja]

for me its defense defense defense before anything else :smart:

N

Defence is the best attack.

 

[DDI]

1) Once backtested, go to every single entry and ask yourself if you wouldve taken that trade had you been there by the pc at the time.

2) Test on different markets and different time frames. A truly robust system should give you a similar equity curve if you backtest it on EURUSD or S&P or oil or even corn or coffee.

I started off a year ago and still havent reached that objective!! Maybe I never will

 

[alphadude]

1) Once backtested, go to every single entry and ask yourself if you wouldve taken that trade had you been there by the pc at the time.

2) Test on different markets and different time frames. A truly robust system should give you a similar equity curve if you backtest it on EURUSD or S&P or oil or even corn or coffee.

I started off a year ago and still havent reached that objective!! Maybe I never will

DDI,

Point (1) is interesting. It does help in understanding your system better.

however; on point (2) it sounds like the Holy Grail. I doubt there is a system that works on every market let alone produces the same equity curve. Even if we consider the turtle system; it will work on commodities as it exhibits momentum more often. However it will not work on S&P500 as the index exhibits more mean reversion than momentum.

So far the systems that I have been trading profitably only works on specific market groups.

 

[alphadude]

Defence is the best attack.

Mike can you be more elaborate?

 

[alphadude]

ALL systems are essentially curve-fitted.

a: You make a general observation about the market.
b: You formalise the observation into a set of rules.
c: You try out those rules, and test using differing parameters giving optimal returns.
d: You re-test against different data, or forward test.

I concur with andrewdbs recommendation of the Way of the Turtle.
The book has a chapter on optimisation.

System designers should spend more time on robustness.
That is, if a system creates a radically different return by changing parameters just slightly, be very wary, it is most likely curve-fitted.

A good system should generate similar returns even if you mess about with the given parameters, within reason.

Trendie; the test you mentioned above is called "Sensitivity Analysis". It helps in finding out if a specific parameter (i.e. moving average period) is robust or not. However it does not help to identify if the whole system is curve fitted or not.

i.e. I can easily design a system that is curve fitted; but all of it's parameters passes the sensitivity analysis.

 

[Adamus]

Trendie; the test you mentioned above is called "Sensitivity Analysis". It helps in finding out if a specific parameter (i.e. moving average period) is robust or not. However it does not help to identify if the whole system is curve fitted or not.

i.e. I can easily design a system that is curve fitted; but all of it's parameters passes the sensitivity analysis.

That is an interesting statement - I agree with Trendie, although I would like to know exactly what you mean. Take for example a system with one parameter only. If you run your sensitivity analysis on your optimisation for that parameter and it shows that it's not sensitive, could you still have made your system curve-fitted?

By curve-fitted I mean obviously optimised to the point where the results are unreproducible.

 

[alphadude]

That is an interesting statement - I agree with Trendie, although I would like to know exactly what you mean. Take for example a system with one parameter only. If you run your sensitivity analysis on your optimisation for that parameter and it shows that it's not sensitive, could you still have made your system curve-fitted?

By curve-fitted I mean obviously optimised to the point where the results are unreproducible.

I do agree that the sensitivity analysis works if a system has only one parameter. Howevever i doubt that such a system exist.

 

[DDI]

DDI,

Point (1) is interesting. It does help in understanding your system better.

however; on point (2) it sounds like the Holy Grail. I doubt there is a system that works on every market let alone produces the same equity curve. Even if we consider the turtle system; it will work on commodities as it exhibits momentum more often. However it will not work on S&P500 as the index exhibits more mean reversion than momentum.

So far the systems that I have been trading profitably only works on specific market groups.

I see your point. I like breakouts and when they work, I get bigger moves on momentum stocks like apple or netflix than fx for example.

However, but still IMO a rovust system (that is based on price patterns) should not have a disastrous equity curve on any liquid market.

Even with your specific market groups, try different time frames and see if the results are similar. If they are, you should be onto a good thing.

 

[trendie]

Trendie; the test you mentioned above is called "Sensitivity Analysis". It helps in finding out if a specific parameter (i.e. moving average period) is robust or not. However it does not help to identify if the whole system is curve fitted or not.

i.e. I can easily design a system that is curve fitted; but all of it's parameters passes the sensitivity analysis.

Never heard of Sensitivty Analysis. Will read up on that.

With regards to curve-fitting, I was under the impression that curve-fitted systems could be detected if small changes in parameter settings resulted in large changes in equity, indicating that the parameter settings didn't define some edge, merely picking up some outlier event that just coincided with that specific setting. Also, sometimes, curve-fitted systems rely on small number of big winning trades amongst the many losing ones. Once the parameter is changed, the many losing ones remain, but the big wins either disappear or replaced with "normal" returns.
I use that concept to ensure I am not chasing dreams. I like to ensure the parameters are robust, so that the many winning trades outweigh the many losing trades, so the "edge" is spread across the entire equity, not just some random fluke event.

A good test of curve-fitting I find is when you get a slew of new systems showing massive profits, just after a major move. You just know they've set the parameter to coincide with the recent outlier.

 

[NVP]

I ask my programmers this question

1) how does the system perform in Flat / Choppy market conditions ?

The answer I need is a very low number of trades activated ...if I dont get that answer we dont even go to the next step

anyone can make money in a trending market and the extent of earnings in these periods is purely system tuning regarding Exits and (multiple) position sizing

N

 

[alphadude]

I ask my programmers this question

1) how does the system perform in Flat / Choppy market conditions ?

The answer I need is a very low number of trades activated ...if I dont get that answer we dont even go to the next step

anyone can make money in a trending market and the extent of earnings in these periods is purely system tuning regarding Exits and (multiple) position sizing

N

NVP: interesting suggestion.

however; what do you do if a system is curve fitted to both flat/choppy and trending market?

i.e. one of my systems was over-optimized for a period of 4 years. The equity curve looks appealing; however it failed in the out-of-sample (or forward testing).

 

[tony1979]

expert system developers,

one of the biggest challenges of building mechanical systems; is whether the system you built has an edge; or simply is curve fitted.

some tools i am using personally are:
- out of sample test
- bootstrap statistics
- forward test (aka paper trading)

what are your thoughts and experiences?

The simpler the system, the better.