Turning the System "Off"

[Chorlton]

Hello All,

Question for all those who trade (or have traded) Mechanical Systems.

At what stage would you turn off a system if it wasn't performing as expected and more to the point how would you know when that time had come?

Some ideas which I'd like to put forward are:

1. The Portfolio £ Capital had dropped by a specific %
2. When the Max DD of the System (as per the backtested results) had been exceeded
3. The Max number of Consecutive losses (as per backtested results) had been exceeded
4. Market Conditions had simply changed. This would be monitored by some form of filter on the system.
5. Portfolio Equity Curve had dropped by a specific %.
6. Mixture of the above
7. Something else.......

All thoughts / comments welcome...

Chorlton

 

[nine]

You could certainly apply 1 and 2 and 3 as crude indicators of potential failure.

My personal favourite is to fully understand how the system makes money, identify the historic market conditions that caused it to lose money, and monitor for a longer period in such circumstances. Once one is spotted you need to determine if its just slightly longer than historic deviation or there has been a change in the longer term market behaviour for some reason.

 

[Chorlton]

You could certainly apply 1 and 2 and 3 as crude indicators of potential failure.

My personal favourite is to fully understand how the system makes money, identify the historic market conditions that caused it to lose money, and monitor for a longer period in such circumstances. Once one is spotted you need to determine if its just slightly longer than historic deviation or there has been a change in the longer term market behaviour for some reason.

Hi Nine,

Thanks for the feedback...

Could you expand on your comments as your approach seems very interesting.

For example, during backtesting, if most of the losing trades happened during a "ranging" Market then would your approach be to calculate the length of the historic Ranging Period and monitor it in real-time to see if similar conditions appeared again?

Once one is spotted you need to determine if its just slightly longer than historic deviation or there has been a change in the longer term market behaviour for some reason.

This sounds a lot easier than it probably is in real-time. How does one assess if Mrkt conditions have REALLY changed? Also, assuming that you can make that decision (and conditions have changed) then what would actually confirm to you that the System should be turned-off. Afterall, if the System was still generating a profit then I guess you would obviously leave it switched on.

Cheers,

Chorlton

 

[nine]

You have the idea. The challenge is to apply it.

I once traded a system that made money from volatility but lost money from trends. It didn't seem to be making as much money as in the past. So was it in danger? Analysis of the volatility suggested that it had declined and until it reverted the risk from trends was high. So I stopped trading it and monitored volatility ... which did not recover for another 18 months.

 

[Chorlton]

You have the idea. The challenge is to apply it.

I once traded a system that made money from volatility but lost money from trends. It didn't seem to be making as much money as in the past. So was it in danger? Analysis of the volatility suggested that it had declined and until it reverted the risk from trends was high. So I stopped trading it and monitored volatility ... which did not recover for another 18 months.

So basically, due to a lack of volatility you simply switched the System off regardless of any other variables?

If so, couldn't you achieve a similar result if for example you monitored the ROC of the Equity Curve??

Actually, thinking about it this wouldn't help with turning the system back on though....

 

[Mathemagician]

There are ways to estimate expected drawdown statistically using resampling techniques. I've done a good bit of work in this area and placed a quick discussion on my website. A good place to go for more detail it he Aronson book. In essence, you can make statements like, "in any 1 year period I expect to see a drawdown of 25% with a 99% probability." If you see a 25% drawdown right out of the gate you can be at least 99% sure the thing is behaving inconsistently with its backtested results. Note that this does not always mean it is broken but often means there is bias in your backtesting process or you've not properly built the system in the first place (dynamic stops, volatility adaptation, etc), which is another topic for another time...

By deciding ahead of time on a confidence level, you're effectively putting a trailing stop on your equity curve whose distance is chosen in a rigorous statistical way. This maximizes the use of available information and minimizes the chances that you halt a good system due to a normal drawdown.

jj

 

[nine]

So basically, due to a lack of volatility you simply switched the System off regardless of any other variables?

If so, couldn't you achieve a similar result if for example you monitored the ROC of the Equity Curve??

Actually, thinking about it this wouldn't help with turning the system back on though....

Yes, you are getting it. You should be monitoring some measure of success or failure. Mathmagician's post is also good.

But. Understanding the systems drivers for profit and failure will give you a number of edges beyond simply realising that its failing. It will stop you trading black boxes. It may provide you an idea of how to change it if conditions change. Etc.

 

[StratOpt]

Regardless of which method that you finally decide to employ as a stop loss to your strategy, this needs to be decided in advance of ever starting to trade the strategy and then it becomes a part of the strategy. You can not waver from this once you decide upon it. It becomes a rule and must be followed. If you do such things as turn the system on and off depending on any criteria then you have in effect simply added a strategy to your strategy and just like any other strategy, if this one is not a winning idea then you are going to have bad results. This new strategy added to the current strategy is also highly prone to being overfitted if you do any sort of optimization to it. This will be directly due to the fact that you won't have many instances of stops and starts within your strategy or else the strategy wasn't likely very good to start with. It is also easy to overfit the idea without ever doing any optimization because you already know the history of the strategy on the market before it and can easily devise a method to avoid where the big losers are. Be careful, be very careful.

 

[Chorlton]

Hi StratOpt,

Regardless of which method that you finally decide to employ as a stop loss to your strategy, this needs to be decided in advance of ever starting to trade the strategy and then it becomes a part of the strategy.You can not waver from this once you decide upon it. It becomes a rule and must be followed. If you do such things as turn the system on and off depending on any criteria then you have in effect simply added a strategy to your strategy and just like any other strategy, if this one is not a winning idea then you are going to have bad results.

I totally agree !!! The intention will be to incorporate this "switch" into the System and then carry out backtesting to see how the System performs with this addition.

Just for clarification, I believe that this "switch" is different to a normal Stop Loss in the sense that a Stop Loss will exit me from a particular trade, whereas this "switch" will prevent me from entering new trades.

This new strategy added to the current strategy is also highly prone to being overfitted if you do any sort of optimization to it. This will be directly due to the fact that you won't have many instances of stops and starts within your strategy or else the strategy wasn't likely very good to start with. It is also easy to overfit the idea without ever doing any optimization because you already know the history of the strategy on the market before it and can easily devise a method to avoid where the big losers are. Be careful, be very careful.

Again, I agree. The "switch" will be basic in its concept and will simply be to stop the System trading under specific conditions.....

Regarding the issue of overfitting / curve fitting, surely this depends on what strategy (as mentioned in Post #1) I use? With the exception of idea #4, none of the others should cause this issue.... IMO....

 

[Tadragh1]

Hi again Chorlton

I tried to give some thoughts on this topic in this thread:
http://www.trade2win.com/boards/mec...lity-market-evolution-towards-holy-grail.html

I actually wanted to post something more specific on the topic, will try later. I definitely like both the statistical approach and the logical approach.

Hi StratOpt,




I totally agree !!! The intention will be to incorporate this "switch" into the System and then carry out backtesting to see how the System performs with this addition.

Just for clarification, I believe that this "switch" is different to a normal Stop Loss in the sense that a Stop Loss will exit me from a particular trade, whereas this "switch" will prevent me from entering new trades.



Again, I agree. The "switch" will be basic in its concept and will simply be to stop the System trading under specific conditions.....

Regarding the issue of overfitting / curve fitting, surely this depends on what strategy (as mentioned in Post #1) I use? With the exception of idea #4, none of the others should cause this issue.... IMO....

 

[Mathemagician]

The statistical approach involves confidence intervals and resampling, or for the brave, parameter estimation. It's not terribly difficult or earth-shattering, but it is sound if done properly.

jj

 

[StratOpt]

Putting in a stop trading is the same thing as a stop loss. It is just a system wide stop loss at a pre-determined level if you use something like a factor beyond the historical max DD. If you use anything similar to a moving average of the equity curve to determine when to start and stop using the model then you have in fact added a strategy to your strategy. If the moving average strategy applied is not effectively a winning idea then you will worsify your position with the strategy and will incur more losses. You are not immune from getting whipsawed there just like you are not immune from it with the regular strategy.

 

[Tadragh1]

In the thread I mentioned before I kind of tried to suggest the perfect approach would be to properly describe the conditions when your system loses money. Then you can run it all the time, with the "turning off" featuer being just a part of the system, like a whipsaw filter.
However, this description of the conditions can probably be done only when your system starts losing money.
So to sum up - my approach would be to use statistical measures to confirm significantly worse performance of the system at present than in the past. Then I would try to determine what is the difference in the market behaviour in the two periods and include a condition in the system, which prevents it from trading when necessary.

 

[StratOpt]

And this is the part where a person can easily manually and mechanically overfit the system. What I see happen with a lot of people is that they will look across their chart and see a few periods of drawdowns on the equity curve. In reality, if you have a strategy that you are trading that is pretty decent, then almost by definition those drawdown periods of concern are going to be minimal in numbers and durations. They can then "identify" what it was that caused the drawdowns, and then easily devise a method to avoid those particular drawdowns. It is very very easy to avoid or minimize the drawdowns in the past. It is not so easy to avoid them in the future. This is where a person has to be extremely careful in what they are doing when they are adding in things to their strategy and then even more careful when they are testing those ideas. This sort of idea has to also be tested in real life on real data, and this would involve a lengthy amount of time of watching the system across various symbols. I can tell you that it is very easy to stop trading a strategy, but it isn't so easy to turn the thing back on after you do stop it.

 

[Chorlton]

And this is the part where a person can easily manually and mechanically overfit the system. What I see happen with a lot of people is that they will look across their chart and see a few periods of drawdowns on the equity curve. In reality, if you have a strategy that you are trading that is pretty decent, then almost by definition those drawdown periods of concern are going to be minimal in numbers and durations. They can then "identify" what it was that caused the drawdowns, and then easily devise a method to avoid those particular drawdowns. It is very very easy to avoid or minimize the drawdowns in the past. It is not so easy to avoid them in the future. This is where a person has to be extremely careful in what they are doing when they are adding in things to their strategy and then even more careful when they are testing those ideas. This sort of idea has to also be tested in real life on real data, and this would involve a lengthy amount of time of watching the system across various symbols. I can tell you that it is very easy to stop trading a strategy, but it isn't so easy to turn the thing back on after you do stop it.

Definately Good Advice....

But (out of interest), surely if whatever mechanism is used to "swich" off the system OR simply to stop a new trade being taken, then this can also be coded and backtested as part of the actual system?

In effect, its just like adding a new Buy Condition to the System's parameters. If over the backtested period this new condition is proven to offer an improvement over the existing system then surely this should offer a good level of confidence.

By then forward-testing using Out-of-Sample data one can see how it really performs.

Regards,

Chorlton

 

[StratOpt]

Yes, you can make it a part of the system and then backtest it. You have then increased the complexity of the system and have reduced the degrees of freedom of the testing correspondingly, but this does not necessarily mean that you have hurt the system. Here is an issue at hand. If you looked across all of your data to develop this idea, then you have introduced a bias to the system by having knowledge of things that work and don't work through certain time frames. You may not even realize that you are doing this, but you will. A good example is that you wonder how your strategy equity curve looks with a moving average on it because it appears that it is fairly smooth and cyclical and almost trend like. So there in excel where you are storing your data you plop on a moving average of 21 day duration and wow it looks pretty darn good. Then you try a 60 day one and hey that looks even better. Every time it drops below it then sure enough you could have avoided a DD, and when it went above then it was really on a run. In effect, you may have just used up all of the data and now you really don't have any out of sample left on the chart. This would mean that you would have to test this on multiple different symbols of which you did not look at before developing and then watch the new additions in a real life walk. You really need to do this real life walk regardless, but using all the data makes this mandatory. You must not develop on the data that is going to be used for the out sample. The problem comes with the fact that you as a person has seen this data, and you run a big risk of introducing a selection bias to the overall testing.

 

[StratOpt]

another example for the recent past ... lets say that you developed a model over the past few years and it has done fairly well up until recently and then it was falling apart. You say heck the volatility is huge now and there are big gaps going on right now so if I put in some filters to cover that then I can fix that problem. So you go back in time and put in those filters. The filters you develop might not be triggered very often in the past so the past history won't look much different, but when you roll into the high volatility period where the strategy had trouble then wow you see a huge improvement. You are now jumping for joy because you have fixed the strategy. This is another example of using up all the data and using your immediate knowledge of what has already happened to "fix" a problem. What you have really done is more likely overfitted the strategy to the recent times, used up all of your out of sample through a selection bias, and now you don't have any data left to walk forward correctly. Now you have a decision to make and that is do you keep on trading the model. I am not saying that everytime you make a change you are going to ruin a strategy. I am only saying that proper testing requires proper thought and proper preparation and an obsession to accuracy. The person most easily fooled into trading on faulty testing is the one who developed the idea mostly because they really want it to work because it is their creation.

 

[Chorlton]

another example for the recent past ... lets say that you developed a model over the past few years and it has done fairly well up until recently and then it was falling apart. You say heck the volatility is huge now and there are big gaps going on right now so if I put in some filters to cover that then I can fix that problem. So you go back in time and put in those filters. The filters you develop might not be triggered very often in the past so the past history won't look much different, but when you roll into the high volatility period where the strategy had trouble then wow you see a huge improvement. You are now jumping for joy because you have fixed the strategy. This is another example of using up all the data and using your immediate knowledge of what has already happened to "fix" a problem. What you have really done is more likely overfitted the strategy to the recent times, used up all of your out of sample through a selection bias, and now you don't have any data left to walk forward correctly. Now you have a decision to make and that is do you keep on trading the model. I am not saying that everytime you make a change you are going to ruin a strategy. I am only saying that proper testing requires proper thought and proper preparation and an obsession to accuracy. The person most easily fooled into trading on faulty testing is the one who developed the idea mostly because they really want it to work because it is their creation.

Hi StratOpt,

Many Thanks for your comments. You seem to talk from experience and I really appreciate your input.

Out of interest, using your example above, how would you overcome the issue of future testing after making changes to the existing System?

The only solution I can think of is that once the new filters had been included, the System would need to be switched-off until sufficient new data had been collected to enable future testing to take place with "clean" data. Depending on the timeframe traded on the System, this could be anything from a number of months to a number of years!!!!


Chorlton

 

[nine]

This is an interesting discussion because it shows so much of the difference between "discretionary" and "system" trading.

I do 3-6 trades every day and actually watch the morning to see how it developed before I trade (my markets open at 9:50 and 10:45 but I don't trade until 12:30). When I trade I take into account variations in the patterns I'm looking for that occurred in the morning to determine how conservative my trigger & price improvement requirements are today. My strategy is very systematic (written down) but the tuning above drives the win rate from 50% to 60% up to 70% to 80%.

If I had a system that was affected by the volatility of the markets I would first look to see if it could be adaptive ... so use ATR or some less crude measure of volatility as it impacts your system to change your parameters. If that couldn't work you seem to have two choices - trade through these periods or switch off at some point. The example I gave above required switching off ... but only once I understood what was "wrong." In that case two "macro" trends had impacted the system - more people were chasing this edge reducing its effectiveness and with the bull market volatility had declined in comparison to trendiness. So I stopped trading it.

 

[StratOpt]

Chorlton,
Once you have made the changes to the system, and if you had used either your knowledge of the data or had actually used up all your data in testing, then you have limited options in what you can do about it. You can do what you describe and just watch the system in real life, but as you note this may very well take years to accomplish and what do you do in the meantime. One thing that you can do is to test this idea out across a wide variety of different symbols some of which you may have never used before to test. You may also decide that what you have done is not going to affect the strategy in real life and trade upon it with a secondary stop loss level that would determine the end of the idea. To a certain extent whenever we are doing strategy developement we are doing so with a knowledge of what the market have been doing and it is very easy to build in a certain bias to the strategy based upon this. This is why it is mandatory that you set aside a certain length of time for watching the strategy work in real life before plopping any money on it and then start slow when you do. This is even after you perform a proper walk forward test that has an adequate efficiency of walk forward result. The ideal thing, IMO, is to have a set of different systems that you are using to trade with currently, and then you are constantly in the development and testing phases with new ideas. This will allow you to easily set that time aside to watch the systems in real time while you are actually trading using other systems that have already been through this process. In time you will run into a tough decision process and that is the fact that you have too many systems and now have to decide which of them to use at any given time, but this is something for another topic. An example of this process is a current set of models that I am using which first went through the walk forward testing process and then I watched them perform in real life for 6 months of time without placing a dime on them although they did very well, and then I started trading upon them, and I am still using them to this day and they are still holding up within their efficiency rate of testing.