Backtesting past and future

[elwardio]

Hi everyone @ t2w.

I am a complete newbie. I have learned a lot from this site.

I have one question that I don't believe I have come across on t2w.

How reliable is backtesting for a system that you have?

Has anyone got a record of a backtest before they started trading and then while they were trading.

ok, two questions!!!

I would really appreciate your answers, comments or suggestions.

Cheers

Andrew

 

[petrovich54]

I am a newbie too, in the sense I didn't trade anything yet. I am only backtesting and trying to figure out strategies to trade.

So far backtesting is no good. Anytime I find something that produces good backtest results for 2006, it fails completely for 2007 and 2008.

I learned here about Monte Carlo and bootstrap tests.

It goes like this: say we have array of 1000 buy or sell signals and corresponding array of 1000 trading outcomes (gain or loss for each signal). We randomly reshaffle the outcomes. With Monte Carlo method, only the order of outcomes is changed. With bootstrap method, outcomes are picked randomly and independently from the original array, so some items will not be picked at all, and some items will be duplicated several times.

If original outcome was from "buy" signal, and in a reshuffled array it corresponds to the "sell" signal, we should change outcome's sign.

Than we compute trading statistics (cumulative gain/loss, sharpe ratio etc) for this reshuffled array. We repeat this numerous times and gather statistics about trading results.

If more than 1% of random arrays have trading results better than our original array, the strategy is not good for trading.

However, even if strategy passes Monte Carlo test, it still can be no good.
In fact, I doubt usefullness of this method. Randomized series share only first-order statistics with the original series. But all the essential strategies depend on high-order statistics, like trends, supports etc. It seems easy to construct a strategy to beat any random series.

 

[simcom]

Another way of testing system robustness is to try it out on different markets and over a variety of timeframes. Although, ultimately there are the same disadvantages of Monte Carlo simulation in the sense that we can never be sure what the future holds - sometimes even random testing isn't enough!

 

[zentrader22]

Monte Carlo Simulation

...and you can use another method which not only "reshuffles" the trades and outcomes (system simulation) but simulates alternative market scenarios based on the original data (data scrambling or data simulation).

Also here you get no 100% reliability - but a better feeling for your system behaviour under diferent market conditions.

I've explained it in a little Traders' article in 2007:
http://www.zentrader.de/mcs_article_traders2007.pdf

bye,
zentrader