Time Periods for Historical Testing

[simcom]

Hi

Not really a 'newbie' as such to trading, but this question is rather basic so thought I would post it here

I have a spreadsheet-based system that is fairly easy for me to test over multiple timeframes and time periods (just a case of inputting different data in High/Low/Close format).

The system has worked very well over the last five or so years, however when I tested it over 20 years using the same parameters the results were much more choppy. It is, however, possible to get the system to work consistently over 20 years/10 years/5 years etc. using the same parameters for each timeframe (although different from those currently used), so would it be more prudent for me to accept a slight decline in performance in the short run by using these, in return for a system which is more robust in the long run?

Or, would it be better to continue to use the more favourable parameters (in the short-run), changing them in the long-term?

Hopefully this is clear, although please let me know if further explanation is needed!

 

[rnicoll]

What data set are you trading on? If it's minute by minute, for example, I'd trade in 3 month slices; optimise the first 3 months, test the next 3 months with those parameters, then repeat forwards. On the other hand, if it's day by day, and your algorithm is breaking over years, it's a bit of a concern...

 

[simcom]

Thanks for your reply. It is daily data - I wouldn't say it breaks down over years, because the current strategy holds up over the last ten years.

However, I was thinking of adapting the strategy to adapt more to market volatility. It is this adaptation which performs well over 20yr+, although marginally (25%ish) less well over the most recent 5yr period. The original method (adapting less to volatility) breaks down prior to about 1999. I'm just not sure whether this is a concern (i.e. is ten years enough?).

 

[simcom]

Sorry to bump this thread, just interested to see if people had any thoughts as I still have doubts myself.

Essentially the question I am asking myself is this: Is it better to sacrifice short-term performance (say, over 3-5 years on daily data) to some extent so that better long-term performance is achieved (10+ years)? Or does the unpredicatable nature of the future make this question totally irrelevant?

 

[donone]

Ask yourself - are you trading 20 years ago or tomorrow?

Market sentiment changes over time.

As do the methods of information retrieval & the typical participants - I think if you have a system which works Now - trade Now. When it stops being profitable - stop trading.

 

[simcom]

Very true. Thanks for your reply.

I suppose I shouldn't get too caught up in the need to cover every possible scenario that could come up, as ultimately this is extremely difficult if not impossible.

 

[the blades]

Sorry to bump this thread, just interested to see if people had any thoughts as I still have doubts myself.

Essentially the question I am asking myself is this: Is it better to sacrifice short-term performance (say, over 3-5 years on daily data) to some extent so that better long-term performance is achieved (10+ years)? Or does the unpredicatable nature of the future make this question totally irrelevant?

chop your data up into chunks (say 4 years), optimise on each chunk and then forward test on the first quarter of the next chunk. Step forward and look at the returns out of sample. Markets change, but your system might be robust to re-optimisation? If so, then use your bets recent parameters. Parameters that suited a long time ago might not now. If your system isn't robust to re-optimisation, be aware of curve fitting as I'm sure you're aware.

I should point out that the above is text book stuff. Though it's the procedure I'm currently following, non of my current portfolio is invested in systems I've developed this way.

Cheers,
UTB

 

[rnicoll]

For reference, working with currencies, I've had good success backtesting two months to generate parameters for the next month.

 

[simcom]

Thank you both for your helpful replies. I think I will carry out the 'chunking' procedure as suggested. Originally I had come to the conclusion that testing a single large period would have the same results, but now I see the relevance of carrying out optimisation for many shorter periods.

I have graphed how total equity differs for changes in parameters between very wide ranges and I get a very smooth hill shaped chart. Hopefully the absence of cliffs and jagged sections means that there is less liklihood of the system having been curve-fitted, but I may have interpreted this incorrectly!

 

[eiffeltower]

Thank you both for your helpful replies. I think I will carry out the 'chunking' procedure as suggested. Originally I had come to the conclusion that testing a single large period would have the same results, but now I see the relevance of carrying out optimisation for many shorter periods.

I have graphed how total equity differs for changes in parameters between very wide ranges and I get a very smooth hill shaped chart. Hopefully the absence of cliffs and jagged sections means that there is less liklihood of the system having been curve-fitted, but I may have interpreted this incorrectly!

Hi simcom, firstly, congratulation on building a profitable system. I myself have been developing trading systems on EOD OHLC data on Excel and Access, doing all kinds of backtesting and so forth, and it feels great to enter a trade knowing that the odds are proven to be in your favour.

I really am not in the school of optimisation (performance seem fantastic on past data, but when applied forward is often a different story. It also takes you away from the fundemental rational that underpins the system, and your own confidence - e.g keep on tweaking the parameters whenever you suffer a series of losing trades.

Are your optimised parameters fairly stable over the various testing periods?
have you front tested the optimised parameters? and come up with reasonable returns each year?

If so then you are very very lucky indeed and good luck!

 

[the blades]

Hi simcom, firstly, congratulation on building a profitable system. I myself have been developing trading systems on EOD OHLC data on Excel and Access, doing all kinds of backtesting and so forth, and it feels great to enter a trade knowing that the odds are proven to be in your favour.

I really am not in the school of optimisation (performance seem fantastic on past data, but when applied forward is often a different story. It also takes you away from the fundemental rational that underpins the system, and your own confidence - e.g keep on tweaking the parameters whenever you suffer a series of losing trades.

Are your optimised parameters fairly stable over the various testing periods?
have you front tested the optimised parameters? and come up with reasonable returns each year?

If so then you are very very lucky indeed and good luck!

Hello eifel,

When you say you're against optimisation it raised a question for me - does your system have parameters and if so, how did you arrive at them?

Optimisation sounds like an awful word that can only lead to curve fitting, but it's also the way in which you can see that your system is stable over a range of parameters, rather than knife edge conditions that will never occur again.

Cheers,
UTB

 

[simcom]

Are your optimised parameters fairly stable over the various testing periods?
have you front tested the optimised parameters? and come up with reasonable returns each year?

If so then you are very very lucky indeed and good luck!

Thanks for your post, eiffeltower. In terms of unknown/future data from the time I tested the method, I have about 9 months of daily trading to go on. I know this isn't anywhere near enough to make a judgement about the effectiveness of the system based on unknowns alone, but an encouraging sign is the fact that the trades during this time seem to very closely match those made on the historical data in terms of win:loss ratio and percentage of winning trades.

Best of luck for your Access/Excel strategies!

 

[simcom]

Hello eifel,

When you say you're against optimisation it raised a question for me - does your system have parameters and if so, how did you arrive at them?

Optimisation sounds like an awful word that can only lead to curve fitting, but it's also the way in which you can see that your system is stable over a range of parameters, rather than knife edge conditions that will never occur again.

Cheers,
UTB

I agree. I hadn't thought about it in these terms before, but an example given by Curtis Faith in 'Way of the Turtle' springs to mind as a good one:

Essentially think of an equity curve which has been generated to show how total returns vary with parameter changes. Optimised parameters represent the top of this 'hill' (if, indeed, it is this shape). Therefore any deviation from these in the future will result in worse performance, but this is still preferable to the deviation from non-optimised parameters. Being on the side of this 'hill', these non-optimised parameters may result in either better or worse future perfomance, but one standard deviation better is likely to still be worse than either of the deviations from the optimised parameters.

Difficult to explain properly without a diagram, but hopefully this is clear. Thought it was good advice.