You've just reminded me that it's time to buy this product:
http://www.scientific-consultants.com/software.html
Nope, not yet. I remembered it cost 60 dollars rather than 160 dollars. I still can't buy it. Too expensive.
What you mentioned actually reminds me more of walk-forward optimization for which I almost bought this tradestation product once:
http://www.rinafinancial.com/Optilogix - removed.asp
But then my friend tried it and it was almost impossible to use (maybe we got it for free on emule actually - I don't remember any more).
However, as I said, Ts Evolve, also a product for tradestation (see first link), does something related to optimization. According to the book by Katz, you feed it all the parameters you know: moving averages, time of the day, range of the day, day of the week, month of the year, etcetera. You feed the software your parameters and it finds the best combination of them all for you, through genetic optimization.
Maybe you should switch to tradestation and we should split the cost of that software.
But I also found that the the same thing can be done, even better, via a brute-force optimization of the same parameters you know, except that you can do it for a lower number of parameter values, because it's doing brute-force optimization rather than genetic (smart) optimization.
I'd say to forget the Rina product instead. It sounds perfect for you:
Are you optimizing a system in TradeStation?
Do you want to periodically adapt you system?
Do you know its possible to use TradeStations optimization capabilities without curve fitting?
Do you want to know what would have happened by changing system inputs based on system performance for a variety of objectives?
Now you can apply walk forward optimization in a seamless environment with TradeStation to minimize the impact of hindsight for more robust system development.
It
sounds, as I said. My friend, who's a programmer, very intelligent and hard-working, said - if I remember correctly - that it's a mess to configure that software. My usual recommendation is to keep things simple, to use the simplest tools, and to do the rest with our own brain, also via guesstimates. For example, for 10 years all I've been using is ts2000i and excel.
By the way, I am not sure you're using the term correctly "walk-foward optimization" (as a synonym for "out-of-sample"), because the way both Rina Systems Inc. and I mean it is that you continually optimize your system's parameters' coefficients as time goes by, and what their software does is tell you if, by doing that, you would have made money. In fact I think this is totally different from the concept of using an "out-of-sample" (even though somewhat related).
Let me show an example about this method, which I have abandoned 8 years ago. It's the concept that by continuously optimizing the parameters of your system you system will be in a better shape to trade the markets. I used to think like this, but not any more.
For example, and this is how my curiosity arose, you find out that if you create and optimize a system based on two moving averages it will work perfectly for a year, and then it won't work anymore.
So you start thinking: are we sure that it doesn't work anymore because it's wrong? Or maybe does it stop working because the markets change?
Then you wonder: what if I re-optimize these two averages every six months? Won't it work at least for the following six months?
That's what the Rina software should be for. To tell you what happens to your systems if you re-optimize them every six months, which would be very difficult to calculate manually.
But now I've abandoned those systems based on two moving averages because I've managed to create systems that, with the same rules/parameters and coefficients, work for the entire in-sample (usually 6 years or a bit less) and out-of-sample (usually 3 years or a bit more).
Oh, here it is:
http://www.amibroker.com/guide/h_walkforward.html
That page explains clearly what the "walk-foward test" is and, implicitly, it shows how it differs from the out-of-sample test:
Walk-forward testing
AmiBroker 5.10 features the automatic Walk-Forward test mode.
The automatic Walk forward test is a system design and validation technique in which you optimize the parameter values on a past segment of market data (”in-sample”), then verify the performance of the system by testing it forward in time on data following the optimization segment (”out-of-sample”). You evaluate the system based on how well it performs on the test data (”out-of-sample”), not the data it was optimized on. The process can be repeated over subsequent time segments. The following illustration shows how the process works.
In fact they're more related to one another than I was saying. In fact the "walk-foward test" is nothing but a repeated "optimization of the in-sample PLUS out-of-sample test" throughout the whole sample subdivided into smaller samples. So you still should not use the term because "walk-foward test" defines a specific method and procedure of "in-sample optimization and out-of-sample testing". What you are referring to is merely doing an "out-of-sample test", don't you agree?
Anyway, if what you do need is precisely "Walk-forward testing", then also Amibroker offers it, besides Rina (used with tradestation). At least that's what they say on that page. I don't trust it, like anything that makes things more complex: there's many dangerous implications in making things as complex as they get with "walk-foward testing".
To recapitulate let me concisely sum things up once again. Walk-forward testing is a form of out-of-sample testing, that divides the sample into many sub-samples and sees the effect of optimization on each one of them.
Wait...
This may be something I was missing out on.
The premise of performing several optimization/tests steps over time is that the recent past is a better foundation for selecting system parameter values than the distant past. We hope is that the parameter values chosen on the optimization segment will be well suited to the market conditions that immediately follow. This may or may not be the case as markets goes through bear/bull cycle, so care should be taken when choosing the length of in-sample period.
Yes and no.
No more than yes, actually.
Not at all, actually.
I was thinking for a second that verifying the effect of optimization throughout the sample would have been a healthy process.
But it is not, in that:
1) As the quote says, you could be picking a period which is too short to comprise all different types of markets. And instead of having a good six years, you will have six smaller useless samples. It's like the scene where Danny De Vito breaks up a cigarette in two, and wants to bet half a cigarette but Jack Nicholson replies that two half cigarettes are useless.
2) It complicates things as far as automation. It is not useful even if you find out that picking the most recent past is more useful than picking the whole past, because its implementation (as fas as my automation of systems at least) is too complex to make it happen anyway. Let alone the databases of system I have, where I have to list together the performance of 71 systems now.
3) It complicates things as far as back-testing, without adding enough benefits in terms of creating a profitable system.