Hakuna Matata
Well-known member
- Messages
- 357
- Likes
- 209
Coming from a data mining background, I do not get what the big deal is with backtesting. I think what is more important is how the strategies you built play in the time after you have built.
For ex: I plan to build a system using machine learning which will tell me to Hold(in my case - no trade to enter), Buy or Sell with fixed risk-reward ratio(1:5). Since I have 3 possible decisions - the system I'm planning to build will be a 3-class classification model. (using R - a link to which someone has posted to this post earlier and PostgreSQL database)
In order for the machine learning algorithm to learn the patterns, I need data (currently in the process of acquiring tick data from Oanda) to train my algorithm.
Say I use data from 01NOV2007-01NOV2012(training period) to learn some patterns/rules. Now, there are 2(actually many more) things to do. I need to understand that my system has learned something novel. This will be accomplished by testing the model performance(in terms of its classification accuracy) on a number of randomly (using bootstrapping) held out records from the training period.
Once satisfied with the performance, the final test(before getting on to the practice account stage) would be to see how it performs on data that my model has not seen before. This is what is called as the out-of-time sample(say data from 02NOV2012-28FEB2013) or simply test data or in terms of trading strategy set-up, forward testing.
This is where I would collect statistics such as Sharpe ratio, draw-down rate, win-loss among a myriad of other things. This is my best estimate of what the future of my trading strategy looks like.
In other words, forward testing is what I will be more concerned with. No point in using a system which uses the same data to learn the patterns and once again the same data to measure its performance.
If these numbers look promising, then start with a practice account and if traded profitably then progress to live account otherwise back to the drawing board.
PS: I was on this forum maybe 3 yrs ago, at that time I was looking into getting into prop trading but I realised I needed a more stable job so ended up joining an Analytics dept. at a bank.
Nice.