Starting Backtesting

[boydd_uk]

I've just read Way of the Turtle.

The one thing the book inspired me to do, which I had always thought would be too much like hard work Is to carry out some testing on historic data. I always thought this was pointless, but I take the view on board of the book that without anything else, what are you going to do??

If anyone could give me tips as a beginner it will be appreciated, specifically:

1. Data. How can I get free data to backtest commodities like Gold/Copper, etc.? Currently I use MLDownloader to download share data, but I don't think this can help with currencies/commodities where the symbol code changes each month. How do you tie up the month by month data to create a single historic data file to test?

2. I'm a newbie when it comes to metastock. Could somebody get me started with a simple backtest say on a system that already exists in metastock?

Thanks for any help!

 

[No one Knows]

Thats one of my favourite books that ive read. One thing i will say about back testing is come up with your trading system then test it not the other way round. You dont want to overfit a strategy to the data because you will get fantastic results in testing and then lose an arm and a leg when you go live.

Mark

 

[boydd_uk]

OK so I downloaded forex and continuous commodity data from http://www.is99.com/disktrading/, which I googled since I didn't get an influx of answers to that question ;-).

I started looking at some test on Gold prices which go back to 1974 (with the data from above). I would of like more data, but not sure if this is available.

I tried the various "expert" systems in metastock with 100K, default 1 unit, took out any interest and found that pretty much all the trend systems performed abysimally? Is this what others have found?

Also the test with Metastock seems a bit primative. I not able to say things like double the number of units if A happens, or more stop up if B happens, etc. Am I missing something, but this seems pretty rubbish for a software package that has been out for ages.

 

[shadowninja]

Yep. I don't doubt that there are winning systems out there but I've backtested loads on MT4 (forex) and found that with a bit of curve fitting some of them can be very profitable... and there's the problem... I had to curve fit the EAs. The proof of the pudding is in forward testing.

 

[meanreversion]

I tried Metastock but found its backtesting to be flaky. For example, it would do "illegal" things if the settings were not correct.. let's say you entered into a trade on the close of a bar with a stop 50 pips away, then if spot had reached that level during the same bar, it would stop you out!! In other words, you would get stopped before you even put the trade on. It's a bug and when I spoke to them, they admitted as much, and advised to always enter on the open of a bar instead (gee thanks).

I use ProRealTime now, which is more robust, looks better and is a little cheaper. The language is also less complicated.

Although Curtis Faith describes the Turtle rules as simple, and they sort of are, programming it is not straightforward and took me a couple of weeks (and I'm fairly fluent at programming). It's certainly worth taking the time to do this, as you can see very quickly which markets it's likely to work on. Moreover, you can see under what kind of conditions you benefit or get hosed.

Currency pairs like EUR/USD respond well to Turtle, equity indices less so. However, you'll want to have at least 10 markets to look at if you're running a full blown trend system.

 

[meanreversion]

Generally speaking the daily data on PRT goes back 30 years or more, which is more than sufficient.

 

[£10kLoser]

Backtesting for years is pointless. What happens if you find that 21 years ago, you would have lost a load of money? Do you really believe you can find a system that would have made money consistently for the last 30 years? The longer you go back the more different conditions become from tomorrow's conditions - which are the only ones that count.

What a waste of time...

 

[meanreversion]

Are you saying there is no point backtesting or that there is no point going back 30 years?

 

[£10kLoser]

There's little advantage backtesting more than a few months (depending on the chart time frame). Thirty years is just ridiculous! For example, many markets were/are quite different after the credit crunch.

 

[meanreversion]

It depends on the time frame. Some longer term systems look at weekly closes, for example. You'd be nuts not to go back at least 20 years (about 1,000 bars). I trade off dailies and tend to scan back 10 years or so, during which time you'll capture all kinds of markets. In fact, sampling back over only the last 5 years covers huge diversity - extreme low vol, extreme high vol, trends, choppiness etc.

 

[zupcon]

Backtesting for years is pointless.

Yet more quality advice from another clueless "Forum Guide" !

you should be throughly ashamed of yourself.

 

[bluewater18]

I've just read Way of the Turtle.

The one thing the book inspired me to do, which I had always thought would be too much like hard work Is to carry out some testing on historic data. I always thought this was pointless, but I take the view on board of the book that without anything else, what are you going to do??

If anyone could give me tips as a beginner it will be appreciated, specifically:

1. Data. How can I get free data to backtest commodities like Gold/Copper, etc.? Currently I use MLDownloader to download share data, but I don't think this can help with currencies/commodities where the symbol code changes each month. How do you tie up the month by month data to create a single historic data file to test?

2. I'm a newbie when it comes to metastock. Could somebody get me started with a simple backtest say on a system that already exists in metastock?

Thanks for any help!

Try www.prorealtime.com

Good luck!

 

[£10kLoser]

Yet more quality advice from another clueless "Forum Guide" !

you should be throughly ashamed of yourself.

This is a meaningless, sarcastic comment unless you back it up with a reason. I explained why it was pointless, so let's hear why you think it has value. Explain why checking against data years out of date in different market conditions serves any purposes to validate a system to be used today.

Only a system that changes and adapts to conditions can show a consistent profit but newcomers want black and white rules to take the place of experience and understanding.

 

[zupcon]

This is a meaningless, sarcastic comment unless you back it up with a reason. I explained why it was pointless, so let's hear why you think it has value. Explain why checking against data years out of date in different market conditions serves any purposes to validate a system to be used today.

Only a system that changes and adapts to conditions can show a consistent profit but newcomers want black and white rules to take the place of experience and understanding.

In simple terms, I'd argue that if your developing and testing strategies then at the very least, you should be splitting historic data into in sample and out of sample data sets. Therefore, the out of sample back test is equally as valid as any forward test on unseen market conditions. I'm in no way suggesting that any type of curve fitting should be applied.

I agree entirely that systems need to adapt, and I agree that simple rules generally dont work well, but thats a completely seperate issue to the validity of backtesting

As always, there's no one correct solution to this issue, and extensive backtesting might not be appropriate in each and every case.However on balance, its an exercise that generally yeilds more benefits than problems

 

[£10kLoser]

As always, there's no one correct solution to this issue, and extensive backtesting might not be appropriate in each and every case.However on balance, its an exercise that generally yeilds more benefits than problems

So, if the backtest shows that 8 years ago, the system would have lost money, what is the benefit? If the backtest shows that 7 years ago, it would have made money, what is the benefit?

I can't see any at all...

 

[zupcon]

So, if the backtest shows that 8 years ago, the system would have lost money, what is the benefit? If the backtest shows that 7 years ago, it would have made money, what is the benefit?

I can't see any at all...

Personally Ive always found it quite useful to understand why a strategy makes money, and why a strategy fails.

In the case of mechanical type strategies, the distribution of returns over time certainly isnt statistically stationary and I'd also suggest that most discretionary methods suffer from the same problem. At least sufficient backtesting provides some insight into both the variability of possible outcomes, and the potential varience in returns.

 

[meanreversion]

I simply fail to see how backtesting has no value. As an analogy, if you were about to buy a house, would you do so with zero knowledge of what house prices had ever done? Would you buy gold without first looking to see what it had been doing? Historical price action is not irrelevant.

 

[bluewater18]

I simply fail to see how backtesting has no value. As an analogy, if you were about to buy a house, would you do so with zero knowledge of what house prices had ever done? Would you buy gold without first looking to see what it had been doing? Historical price action is not irrelevant.

Backtesting appears everywhere. Weather forcasting, travel times, mortality rates, responses to advertising etc. The list goes on. It helps people and companies to know what they are likely to encounter in the future, based on past events.

I see no reason not to employ a similar systematic approach to backtesting my theories to identify pockets of non random price behavior in the marketplace.

Hey it's almost 8.45am. I usually have a coffee around this time. Gotta go.

 

[Xeno]

Only a system that changes and adapts to conditions can show a consistent profit but newcomers want black and white rules to take the place of experience and understanding.

Oh dear, you want to know why multi year backtesting is useful, then you give the reason yourself.

Clearly, if you have a strategy that adapts to conditions you'd want to test it against as many types of market conditions as necessary. Similarly, if you have an intra day pattern system, you'd want to test it against many days. If it trades one trade every two days then two years is 250 trades. That seems like a reasonable sample size. A few months data would be 40 or so trades. I wouldn't be happy with so few results.

Yet more examples. If you test any indices trading for the last year, you'll do very well on long trades. Double the testing period, and you suddenly have 12 months when short trades did very well. That will make a huge difference to your strategy. Finally, to help inform you about curve fitting when testing intra day strategies, you can run optomised parameters against out of sample historic data. i.e. optimise over the last year and try it against 2 years ago.

I think you're confusing the fact that markets never repeat themselves exactly with the fact that patterns repeat regularly, on all timescales.