Backtesting - is it a valid concept ?

[Glenn]

Hi Mr Cassadra
Agree with your thoughts.
I've continued experimenting with one system which was tested since June 1998. This covers quite a range of market types.
One modification I have employed is in respect of the Equity curve itself. By applying a simple 30 day moving average to the Equity curve it gives a signal when to stop trading the system (because it has gone into a losing phase) and when to start again.
6 months on and it's still working ok, including a 40-day 'stay out' period.
The system itself is designed to take account or recent volatility (or the opposite) and respond accordingly.
This way it tries to include current conditions and data in the math model, rather than just relying on an indicator etc.

Have you tried what I described as 'forward testing' ?
e.g. Test and refine a system over 3 years and then run it forward over the next 2 years to see if it still works. (These are just example numbers, - the longer the better in both cases imo.)

Glenn

 

[georgek]

Feel So Lucky



I'm into FUTURES DAY TRADING , E-minis.

I've traded for a couple of months,like everyone,lost money,no system,no method,no discipline.
It's been 2 or 3 months now,that i started developping a SYSTEM.Based on MFL(Metastock Formula Language) and backtested on 5-min intraday data.

I can't say more than i 've seen.

When i reached at a point where i thought my system was good (always testing for a month or 2 of intraday data) i went back to the past and tested it LIKE IT WAS(with no changes) for another different period of time.

The results were bad.There was no profit,there was loss.I Optimized my system for this period,and the parameters changed,now there was profit.

In the next days(the following week) i'm planning to start trading with that system.
What i'm hoping?
What i'm hoping is : Trade with the last good system parameters that i have.Trade for a week.
Have profit similar to the one i had for the past.Re-calculate my parameters at Friday and continue trading at Monday with the new ones.
Then wait till 1 month (22 trading days) finishes and then see if i'm positive or not.

I can't say that i'm optimistic , i wasnt before, and now,with what i've read , i'm more scared of what could happen.I trust my system though.I trust what i've created.I hope it will go well in trendless and trend days..

I'm looking forward to your comments

ps:feel lucky because i found this forum,especially this topic.

 

[Glenn]

Hi Georgek
In my opinion :-

Optimisation is a problem. I think it is better to have a system which is not optimised.
Optimising the past does not mean it will work in the future.
If you find a system which works, you also need a way of knowing that is has stopped working. That is why I have been using the 30-day average on the equity curve.

Backtesting needs a lot of historical data, not just a couple of months.

You say "I trust my system though. I trust what i've created."
Why do you trust it if it doesn't work ?

Glenn

 

[theknifemac]

Hi Georgek
In my opinion :-

Optimisation is a problem. I think it is better to have a system which is not optimised.
Optimising the past does not mean it will work in the future.
If you find a system which works, you also need a way of knowing that is has stopped working. That is why I have been using the 30-day average on the equity curve.

Backtesting needs a lot of historical data, not just a couple of months.

You say "I trust my system though. I trust what i've created."
Why do you trust it if it doesn't work ?

Glenn

I think optimisation is fine AS LONG AS the optimised parameters picked work fine when they are changed - i.e. if you look at your optimisation results as a surface you want the point you have picked to be in the centre of a plateau rather than a peak. This increases the chances of you having found an edge which will persist in the future rather than a curve fitted system which will perform terribly in the future.

Cheers

Stew

 

[georgek]

Thank you both for your answers.

Well,the thing is , i trust my system because it really has some profit in a 2-month period.

2 months of INTRADAY data is fine , thinking that 2 months of 24/24 ,5-min candles..
are a lot of candles!

This is a complex system,changing methods that it enters markets (depending on trend - trendless days)
and i've optimized every single period and line of it...

Thats why i trust it
I'm sure thinking of what theknifemack told....My optimizations maybe need to be chosen more carefully.

Still,i am not optimistic.

 

[sidinuk]

The market characteristics change dramatically over time. Even with intraday data you need to look at at least 2 years to make the backtest worthwhile.

 

[frugi]

Perhaps this problem can be mitigated by diversification?

Run several robust systems concurrently for equal stakes. Perhaps each trades a different instrument by a different method on a different timescale from the others.

If any system enjoys a number of consecutive wins equal to the "max consecutive wins" indicated in the backtest, then discard the system until it has suffered a similar number of consecutive losers. While a system is inactive one could add to the stakes of the others to compensate.

This goes against the conventional wisdom of running your winners and cutting your losers, but, as most successful systems are trend following, a run of consecutive losers would suggest prolonged consolidation after which you could reasonably expect a nice big trend, and vice versa. Though remember that markets consolidate for much longer than they trend, on average.

 

[georgek]

well i can say i'm a little confused by this information mr frugi!
i'll try to read again and again more carefully

 

[Robb]

This article, written by Curtis Faith one of the original turtle traders, offers a view that may contribute to this discussion -

http://www.turtletradingsoftware.com/articles/optimization_paradox.htm

 

[georgek]

VERY INTERESTING ARTICLE!

i can say i am a little disappointed about backtesting though .

 

[TheBramble]

i can say i am a little disappointed about backtesting though .

George, I think we all are when we discover the limitations (pointlessness?) of back-tetsing.

When you realise you're fitting your 'systems' to what was, in the hope you'll capture what is to be...

There's a benefit in looking to the past to decide how the future may pan out (as in not making the same mistake twice), but traders tend to bend this approach right back on itself.

 

[theknifemac]

George, I think we all are when we discover the limitations (pointlessness?) of back-tetsing.

When you realise you're fitting your 'systems' to what was, in the hope you'll capture what is to be...

There's a benefit in looking to the past to decide how the future may pan out (as in not making the same mistake twice), but traders tend to bend this approach right back on itself.

I understand what Curtis is saying but I think it still makes sense to perform backtesting - a system which backtests badly is likely to perform badly, unfortunately the converse is not necessarily true. When looking at the back test results you should see what happens when the parameters are changed slightly, if the results are consistent then the system is unlikely to be over optimized.

As Curtis puts it "just because optimization results in tests that overstate likely future results, this does nto mean that optimization should not be done". It means that the best set of parameters for the system are not necessarily those which produces the highest profit / best statistics, instead you are looking for parameters which put you in a flat area of the optimization results rather than at the top of a peak.

Stew

 

[georgek]

I understand what Curtis is saying but I think it still makes sense to perform backtesting - a system which backtests badly is likely to perform badly, unfortunately the converse is not necessarily true. When looking at the back test results you should see what happens when the parameters are changed slightly, if the results are consistent then the system is unlikely to be over optimized.

thats the point.


Curtis , my english is not so good and i cannot understand 100% what you'are saying..

 

[theknifemac]

thats the point.


Curtis , my english is not so good and i cannot understand 100% what you'are saying..

Don't worry George, your English is a lot better than my Greek ! And congratulations on the football. Yamas !

Stew

 

[georgek]

Haha thanks ! we were great,everyone's crazy about this success here !...

 

[mr_cassandra]

I have and I do

You can get many widely diverse opinions about back-testing but many times the person is just quoting some article they read or commentator they listened to.

I invest with real money using a program I wrote myself over the last 4 years, called the mvp signal system. It contains data and back-testing going back into 1995. During this period I've had all sorts of feedback pro & con but only about 1 in 20 responders had any research background of their own, ie they actually went and crunched large amounts of numbers on their own.
In my opinion, from having been at this awhile, three things are crucial for back-testing to be of value.
1. The back test period should include widely varied types of markets from flat to bull run to a bear market. As regards a bull run, it would be even better if it included a 'bubble run' or as Mr Greenspan puts it, - euphoria. During the past 8 years all of that has been seen in the U.S. Market.
2. The settings, or 'triggers' in the system should be tuned 'down' to the mildest possible parameters.
3. You also have to be careful that the back-tested results have not caused or included some wild trade(s) which make it look great but were probably anomalies.

Last but not least, the system must act the same way and produce the same results when used in real-time. That is the stage I'm currently at. My website explains in more detail and I post the signals in real-time at clearstation.com (my prior posts contain links to these)

Another site doing a system (succesfully) is www.vtoreport.com - 5 day rsi system.

Regards, Steve

I wonder what others think about the concept of backtesting ?
One the one hand it perports to find the best set of parameters for a given method and thereby tune it to the period tested.
If the results are good, then these parameters can then be used with some (?) confidence for live trades.
On the other hand one could argue that backtesting is just curve fitting and therefore simply 20:20 hindsight.

Seems to me that for a backtest to have any validity, it must produce a smoothly increasing equity (profit) curve with no wild girations. This at least demonstrates that the method and parameters found worked fairly consistently over the period tested.
Of course a lot depends on the period(s) selected.
Again to have any validity, I think that a test must either be run over a considerable time period, embracing many different market conditions and yet still remaining consistent, or it must be run on selected market conditions with a view to using the optimised parameters from the test in that type of market.

To disprove the theory of curve fitting, one suggestion is to take a period of time, say 5 years, and run your test over the first half of the data to get some results.
These results can then be run over the second half of the data as if it were the unknown future. i.e. a kind of Forward Testing.

One thing's for sure, the Software vendors must love things like backtesting. What an opportunity to develop and sell a nice technical product, using the math co-processor and the disk drive of your PC to the full. A nice 'gadget'.
Lovely jubbly - at least for the vendors.

Enough from me. What do others think? In particular anyone who has used a backtested system in the real market sucessfully.

Glenn

 

[mr_cassandra]

re forward testing

This is a tough one. I'm on version 21.C as we speak and all of its code is a collection of if-and-or but logic which has been evolving and being improved since day 1. I understand the gist of your thought and agree, but would not know how to apply it.
One other observation, any code change I ever make to the program is always copied back to day one to look for what I call 'bogeymen'. If it can't bring a steady and/or calm reaction to the entire 8 years, its trashed.

Hi Mr Cassadra
Agree with your thoughts.
I've continued experimenting with one system which was tested since June 1998. This covers quite a range of market types.
One modification I have employed is in respect of the Equity curve itself. By applying a simple 30 day moving average to the Equity curve it gives a signal when to stop trading the system (because it has gone into a losing phase) and when to start again.
6 months on and it's still working ok, including a 40-day 'stay out' period.
The system itself is designed to take account or recent volatility (or the opposite) and respond accordingly.
This way it tries to include current conditions and data in the math model, rather than just relying on an indicator etc.

Have you tried what I described as 'forward testing' ?
e.g. Test and refine a system over 3 years and then run it forward over the next 2 years to see if it still works. (These are just example numbers, - the longer the better in both cases imo.)

Glenn

 

[Glenn]

This is a tough one. I'm on version 21.C as we speak and all of its code is a collection of if-and-or but logic which has been evolving and being improved since day 1. I understand the gist of your thought and agree, but would not know how to apply it.
One other observation, any code change I ever make to the program is always copied back to day one to look for what I call 'bogeymen'. If it can't bring a steady and/or calm reaction to the entire 8 years, its trashed.

Re Forward Testing

All I would suggest in your case is that you backtest over the first 5 years to get the system parametrs tuned, and then you use those parameters over the next 3 years data and see how the system performs.
OK this does not utilise the whole 8 year period, and it precedes 9/11/01, however it would be interesting to exclude that particular unfortunate event and see how the system coped with it as well as the remainder of the bear market which followed and the bull run since.
Glenn

 

[mr_cassandra]

Many ways to show it does

The market is a large herd of cattle, stampeding back and forth. Their numbers reach repeating extremes, particularly on the down side. It takes a lot of work to go diligence this. Long ago I ran barrons stats on microfiche and entered into a laptop to define net new lows, year after year.

You'll find that most naysayers of such concepts have not actually done the research themselves, rather are relying on books, articles or comentators they have read.

Glenn,

Before you ask about back testing being of any value , you have to ask yourself another more important question..
DOES PAST CORRELATE tO FUTURE ?

You can only have three answers .


1) yes it does..
2) No it does not
3) it does to some degree
if you think number 3 is the answer then your back testing will be effective to the same degree ..

 

[Naz]

Never mind the past look to the future.If you see it take, manage it, trade it.

To many people get analysis paralysis and hence get no where.

Of course this is just a personal opinion, others will probably think differently.