The Tenets of a Successful, Backtested, Mechanical Trading System

I find a lot of posts in this forum are asking questions rather than providing solutions. So, I'm going to provide what I believe to be the major important tenets of a mechanical trading system. I see a lot of confusion out there about what makes a mechanical trading system successful. This post should clarify what a system designer should aim for and what he/she should believe about trading.

Major tenets and beliefs:
1. Human traders or discretionary traders lose trades because of fear and greed.
2. Human traders or discretionary traders cannot maintain the necessary discipline to trade consistently forever.
3. Market movements of all kinds (price, volume, volatility) are 99.99% determined by human factors rather than mechanical systems.
4. Every profitable trade in history will be repeated in the future.

These are beliefs, which I feel are very close to reality because I have tried discretionary trading, and I also design mechanical systems. I have been lurking on T2W for about a year, and I've designed two mechanical trading systems. Full disclosure: I have not made any real money with these systems yet, but I feel that I will in the next few months once I move away from paper trading. I am not fully qualified to tell you what works in a real-money system. However, I am qualified to tell this forum how to design a system that backtests very successfully.

Just starting out last year with backtesting software, I could barely get an upward sloping equity curve in simulation. Now, I've got an upward sloping equity curve with very low drawdown over a span of 20 years. Some simple stats I can give you are my backtested cummulative annual return (53%), and the maximum system drawdown (28%). Both of these stats are over the time period from January 1, 1992 to June 29, 2012. I've learned so much in how to design a successful system in backtesting software over the past year. So, I am qualified to tell this forum how to design a successful system in backtesting software.

Key Success Factors in designing a successful mechanical trading system in backtesting software:
1. Performance goals
2. Accurate and comprehensive data
3. Positive expectation
4. Risk management
5. Time frame
6. Optimization

If you can understand and do well at each of these areas, you will design a profitable mechanical trading system in simulation. Here's more elaboration on each of these key success factors:

1. Performance goals: Decide for yourself BEFORE backtesting anything what drawdown you can handle. Imagine you had a working, real-money system. Then imagine that next week your system lost 25% of your money. Would you break down and cry? Would you stop trading out of fear? If you feel a cringe at losing that much money, try 20% or 15%. If you're okay with 25%, what about 30% or 35%? Another stat you should define is time in a drawdown. Let's say you recently had a good month of trading, and your account value is the highest it has every been. Feeling good! Then, let's say your system loses 10% next week and doesn't recover for 6 months. Could you handle that? If you trade for clients, could your clients handle that? What about 5 months? The last stat to define is your desired cumulative annual return. Would you be happy with 25% per year? If so, what about 20%? Keep going down until you find a number that sounds unacceptable to you. These three statistics are your performance goals. Design your system to achieve them. Change your system if you feel it doesn't meet these goals. Be serious and realistic about this! Don't be silly and choose a 5% drawdown, 2-week time in drawdown, and a 300% cumulative annual return.

2. Accurate and comprehensive data: Your data source should provide data for all the stocks you want to trade. Also, it must provide data for all de-listed stocks you would have traded. Make sure you go through some stocks to make sure that stock splits were handled correctly. Some backtesting software and data sources do not account for dividends. It is up to you to decide whether that invalidates your backtest results or not. A system that has an average trade length of a few days should be fine with not accounting for dividends. A system that holds positions for weeks or months should somehow account for dividends. Remember that short positions pay out dividends rather than receive them.

3. Positive expectation: It is very very important that your trading rules have a positive expected value. Mathematically, this means (win rate)*(win amount percentage) - (loss rate)*(loss amount percentage) > 0. For example, let's say you design a fictitious system that buys on a moving average crossover and sells 4 days later. Your system wins 58% of the time and loses 42% of the time. Your winning trades make 3.5% on average and your losing trades lose 3.7% on average. Your expectation is 0.58*0.035 - 0.42*0.037 = 0.00476, which is greater than zero. This means that you can expect to make 0.476% on average on every trade you enter in the future (Tenet 4 above). It is very difficult to design a profitable trading system with a negative expectation.

4. Risk Management: Risk is the potential that a particular trade you enter will results in a loss. Every trade you enter has a chance of losing. Some trades have a bigger potential loss than others. You can find out some factors that produce bigger winners and smaller losers. Perhaps stocks with a very high average true range win big with your system? If so, bet more on those positions and less on others. Also, consider how big your stop losses and profit targets are if you use them. There are many position sizing methods to use here, but the main ones are "fixed risk" and "fixed fractional" position sizing. The Kelly formula will give you a decent starting point for position sizing. Google that.

5. Time frame: Just about every random mechanical system ever created is very successful over a short time frame like a month or a year. However, you have to know when to start trading that system and when to stop. A short-only, trend-following system would have been great during the time frame of September 2008 to March 2009! How do you know when to stop trading the system? It's very hard to predict turning points in the market like March 9, 2009. It's much simpler if you design a system that performs well or at least okay during up, down, volatile, and quiet markets. It's okay if you can get your system to make hordes of money in some types of markets and just limit losses other types of markets. The key statistic is your expectation over a long period of time (like 10-20 years). Hopefully, it is positive. If you have positive expectation, you can adjust your risk management to achieve your performance goals.

6. Optimization: Run your backtest over your selected time frame with a range of different parameters. For example, going back to our moving average crossover system, test over different periods for the moving average. You can test a range of values for the periods of any indicator, the stop loss amount, the fixed risk amount, the number of days to stay in a position, etc. Any parameter that is numeric can be optimized. In your optimization, you want to make sure that your system makes money over a wide range of values for each of these parameters so that your simulation results are robust. Once you confirm that, choose the parameter values that produce the best performance (e.g. highest annual return, lowest drawdown, highest win rate, etc.).

I hope this helps some people and stirs some discussion. Thanks for reading!

Great post - a beacon post - ie it stands out in amongst all the other dross here - I just hope it illicits the intelligent discussion it deserves and all participants of the thread contribute intelligently and move the discussion on in a positive manner and in so doing may learn something/get value from it - you included. Of your 4 key tenets:

Major tenets and beliefs:
1. Human traders or discretionary traders lose trades because of fear and greed.
2. Human traders or discretionary traders cannot maintain the necessary discipline to trade consistently forever.
3. Market movements of all kinds (price, volume, volatility) are 99.99% determined by human factors rather than mechanical systems.
4. Every profitable trade in history will be repeated in the future.


I have no doubt that 1 is true, 2 is probably true and 4 is true but depending on the market/instruments - 3 may be less true in that HFT/algorithms may play a bigger part than you think ?

Best of luck with your ongoing efforts - such a methodical well thought out approach and effort certainly deserves rewarding by the market but this alas does not always follow.

G/L

Interesting post, and as bbmac says, it's good to see some effort going in.

I think there are more reasons than just fear and greed in 1. I've had losing trades because my concentration was lacking, because I was ill, because I was agitated by someone else, because I misread things and became confused etc. The point you made still stands though with regard to errors in discretionary trading.

I think it's a very sensible approach you're taking. I would also add though, that there are also some drawbacks with mechanical trading.

Excellent post by wheresthejelly. With a mechanical system that has positive expectation you can make a lot of money provided -- and this is the important thing: provided you stick religiously to your rules. Sticking to the rules is the hard part!

I might be missing something, but I saw no mention of forward testing - or out of sample. Getting a fit over in-sample data is fairly easy, the optimisation just curve fits. Getting similar performance out-of-sample is much more difficult. I'd be interested to know how you forward test - what % of data do you reserve for this and at what point do you say a forward test is successful. Really good post - thanks.

wheresthejelly said:

Major tenets and beliefs:
1. Human traders or discretionary traders lose trades because of fear and greed.

Click to expand...

Add to that: Inexperience, lack of understanding, incompetence

wheresthejelly said:

2. Human traders or discretionary traders cannot maintain the necessary discipline to trade consistently forever.

Click to expand...

I’d say most give up or go broke before they even get anywhere near that point. I would also say that mechanical system traders cannot resist the temptation of consistently modifying and/or interfering with their system whenever they experience a string of losses. The idea that mechanical trading systems eliminate the problems of emotions and discipline from trading will be just a fantasy for most people.

wheresthejelly said:

3. Market movements of all kinds (price, volume, volatility) are 99.99% determined by human factors rather than mechanical systems.

Click to expand...

Yes, people seeking profit makes a market.

wheresthejelly said:

4. Every profitable trade in history will be repeated in the future.

Click to expand...

I’d say the conditions that lead to a profitable trade will be repeated time and time again. So yes, I agree.

bbmac said:

Great post - a beacon post - ie it stands out in amongst all the other dross here - I just hope it illicits the intelligent discussion it deserves and all participants of the thread contribute intelligently and move the discussion on in a positive manner and in so doing may learn something/get value from it - you included.

Click to expand...

Thanks bbmac! I think we've got a good thread going so far.

Shakone said:

I think there are more reasons than just fear and greed in 1. I've had losing trades because my concentration was lacking, because I was ill, because I was agitated by someone else, because I misread things and became confused etc. The point you made still stands though with regard to errors in discretionary trading.

Click to expand...

I'm starting to think Tenet #1 and #2 should be combined. Basically, humans are fallible and inconsistent. Mechanical systems have trustworthy consistency and do not succumb to any emotions.

doh said:

I might be missing something, but I saw no mention of forward testing - or out of sample. Getting a fit over in-sample data is fairly easy, the optimisation just curve fits. Getting similar performance out-of-sample is much more difficult. I'd be interested to know how you forward test - what % of data do you reserve for this and at what point do you say a forward test is successful. Really good post - thanks.

Click to expand...

This is a GREAT point that I should have brought up. It's all about meeting your performance goals, and I think that a system designer may choose one of two routes to accomplish multiple-time-frame testing:

1. Design a system that meets his/her performance goals over a very long time period that includes bull, bear, volatile, and quiet markets (like the past 20 years). If the designer is happy with the simulated equity curve, then there is no need for forward testing.

2. Design a system that meets his/her performance goals over a specific type of market (e.g. the US bear market from 2000-2003). Then, the designer should forward-test into the next few years (2004-2007) with parameters (i.e. MACD bars, RSI threshold, stop loss amount, etc) that were successful from 2000-2003. Then, optimize the parameters across the time period 2003-2007, and use these values for the next time period from 2008-2011. Once the forward simulation reaches the end, the designer looks at the equity curve and drawdown to see if it meets the performance goals. If not, tweak the system.

I have no hard criteria to say whether a forward test was successful other than comparing the simulated performance with the performance goals.

0007 said:

Excellent post by wheresthejelly. With a mechanical system that has positive expectation you can make a lot of money provided -- and this is the important thing: provided you stick religiously to your rules. Sticking to the rules is the hard part!

Click to expand...

That is so true! I keep getting the engineer's bug to want to tweak something. I should just sit on my hands sometimes. :-D

new_trader said:

I’d say most give up or go broke before they even get anywhere near that point. I would also say that mechanical system traders cannot resist the temptation of consistently modifying and/or interfering with their system whenever they experience a string of losses. The idea that mechanical trading systems eliminate the problems of emotions and discipline from trading will be just a fantasy for most people.

Click to expand...

You are so right with the traders of mechanical systems. Our human nature can affect the system because we control the system's on/off switch.

The way to remedy this (from my own experience paper trading my system for the past 3 months) is to have a clear focus on your performance goals and set a stopping point for yourself. Here's what I do when I'm faced with a string of losses:

1. Ask myself, "Is there a bug in my code or did something change that I'm not aware of?" If so, fix it NOW.
2. Ask myself, "Does a simulation of my system show the same drawdown that I just experienced?" If so, then the system is behaving normally. If not, investigate.
3. Ask myself, "Have I experienced a 25% drawdown yet?" This is my hard limit for a drawdown. I will stop trading, lower the position sizes, and start over if this ever happens.

If I follow this, then I should be a consistent operator of a mechanical system.

Oh dear, someone is about to be taught a very expensive lesson

the hare said:

Oh dear, someone is about to be taught a very expensive lesson

Click to expand...

Why?

Wheresthejelly, your post was quite interesting.

I'd like to offer up a bit different perspective on the elephant of "mechanical trading systems".

I 'd like to address what I view as "fundamental tenets":

- Mechanical trading systems are necessary to enable valid testing and measurement.

There are two reasons for this. First, if there are discretionary components, we won't apply those in a consistent manner over time. Effectively, that means we are never testing or measuring or trading the same system. It is shifing, and all it's fundamental metrics (assuming any trading system truly has them, a debatable point itself) shift with with those changes. The second is the simple problem of scale: we want to measure a system over LOTS of market data. Impossible to do with a system with any diescretionary component.

- It takes large amounts of test results to draw any tenative conclusions.

I personally have tested my most important system on over 60,000 trades (all "forward" after initial development, across different stock universes and different timeframes). And I don't think that's anywhere close to "enough". I take my experience here not from the world of trading, but from poker, where it is well known that at only 60,000 hands, we have some information on our results, but it is definitely not yet a stable set of metrics. 20,000 is completely suspect and while it gives a general (likely) indication about profitability, the variance is far too high to draw substantial conclusions. There's lots of material in the twoplustwo poker forums on this subject, and as a poker player with about one hundred thousand hands, I can see the variance that these assessments come from.

- If a trading system doesn't work across "all" (or at least "many" to "most") markets, it doesn't work.

Probably the most contentious point I'll make. It is a bit far reaching I realize, but the simple logic is this: if there are markets it DOESN'T work for, what to stop the market it does work for changing in attributes (likely soon after you start deploying real money on it) to the one it doesn't work on? I deployed a system that behaved incredibly well on 4 years of British Poind data, but within a week of deploying it, the nature of that market that the system fundamentally depended on stopped and changed. It's perverse, and it will happen. I also realize there are limits here, because any mechanical system can be destroyed by "creating a market" that uses knowledge of how the system operates to make it fail, and in theory there are are infinite markets and market behaviors, so by definition, a system can't work in all markets. But hopefully you get my point.

- A percentage point of positive expectation is excellent. More is exceptional.

This isn't always well understood, for many reasons. If you've got a system that consistently clocks over 2% average return per trade, you can with no question whatsoever be completely independently wealthy quite quickly. It's hard to achieve, and even 1% is very difficult (with consistency). Don't let "big winner" systems confuse you to believe they are big average winners, they are not.

- A trading system must have as a foundation a per trade positive expectation. Without that, you have nothing of value.

I believe you had a typo or unintended misstatement when you said "it is very difficult to design a profitable trading system with a negative expectation." Incorrect. It is impossible. If you have a negative expectation on a per trade basis, you are doomed. The more you trade it, the more doomed you are (to lose money overall). Nothing in any of the areas you cite (performance goals, risk management, time frame, optimization) or others (betting strategy usally come up right here) will change the fact that it is not profitable. Sure, you can run it for a bit and "get lucky", but "profitability" is usually used to mean "profitable in the long term and on average". In one sense what I'm pointing out is profitability = expectation. The more important point is a positive expectation is the absolute must starting point, or you really just have an ability to give money to others, which is much easier to accomplish than through operating a trading system! That said...

- A trading system only starts with a positive expectation per trade mechanic. There are many additional elements (or practices) that must be addressed in a complete and pragmatic (reasonable to trade our real money) trading system.

What do I mean by that? Diversification (for risk management, variance reduction and higher probability of achieving the average expectation), trading fund allocation into trades, compounding method, commission management, special circumstance management, and others all need to be address in a truly complete trading system. You cited some of these areas, but not all, and some such as "optimization" and "measurement" are just means to the end, not fundamental attributes of a final system.

- Multiple simultaneous trade opportunities must be generated by the system.

A complete system has to address how you take a very small edge (say, 1%) and make consistent money out of that. If the system only gives us one trade at a time (or worse, no trades at all for periods of time), then we cannot "take it to the bank", that is, run it fast and hard with all our money to generate profits. We needs "lots" of system trade opportunities available most of the time, so we can both get all our funds working, and get it working in a way that reduces variance and risk and drives results towards the system's expected value with a greater probability.

- Diversification across multiple trades is paramount; variance is an evil.

There is no trading system that is going to give such wonderful odds that we can afford to take the risk of putting all our trading money into a single trade. Again, 1% average profit per trade is "excellent". So we have to design a system that gives us the same kind of diversification the casino gets in how players play their small edge games. Their biggest fear of the casino (aside from the cheater I suppose) is the gambler who comes in with a giant wad of money and makes one bet! (One reason they have betting size limits.) They make their consistent profits on the law of large numbers. That same giant wad of money bet instead in very small increments assures them of steady profits, every day. Similarly, as "the house" executing a positive expectation trading system with a very small positive expectation, we cannot entertain the "one big bet" approach, we have to use the casino method of lots of smaller trades. Our trading system needs to enable us to do so. The "perfect entry" system that finds one trade a week for example is not of much pragmatic value to us. The goal here is reducing variance, and we acknowledge that doing so will cost us some in expectation due to the dreaded commissions. But it is an absolutely required tradeoff.

- Overrides needs to be addressed

This one runs contrary to the first rule: it must be mechanical, so it's stable and we can test it. However, when we move from the world of testing to the real world of trading, we must have pragmatic overrides. They take lots of forms, both common and rare. For example, what do we do when we are on vacation? How about when war breaks out or terrorist attacks occur? Or flash crashes? I use the term "stochastic shock" as a general term for "the market goes totally nuts and executes a 3 or 4 or 5 sigma event". It happens. We have to get real about this kind of stuff, or it will get real on us, and we'll find our trading account, so wonderfully nurtured over the last month or year or years, suddenly devastated by giant losses. How about after stable trade results, we suddenly get a spate of losers, beyond anything we've ever seen in testing. A complete trading system that we are going to deploy in the real world with our real money needs to explicitly address all these kinds of areas, as opposed to being "theoretical" at just the level of a simple set of mechanical rules that get run "forever".

- The system must be feasible to operate reliably

This varies for everyone of course, but there needs to be a basic "impedence match" between the system operation requirements and what we the executionist of the system can afford to spend on it. This may be a time issue, or if it's a totally mechanical system that we could implement in a 100% automated manner (direct submission of orders etc.), then it's a cost issue of such a set up, and the cost and effort to get it extremely reliable and fail-save and fail-soft. Without total automation, the largest issue is time required, which is primarily driven by the timeframe of the trading system.

I'm sure I've missed some...but that's what comes to mind right off.

I hope this sheds some additional light on this exceptional subject!

-Kevin

wheresthejelly said:

...........

You are so right with the traders of mechanical systems. Our human nature can affect the system because we control the system's on/off switch.

The way to remedy this (from my own experience paper trading my system for the past 3 months) is to have a clear focus on your performance goals and set a stopping point for yourself. Here's what I do when I'm faced with a string of losses:

1. Ask myself, "Is there a bug in my code or did something change that I'm not aware of?" If so, fix it NOW.
2. Ask myself, "Does a simulation of my system show the same drawdown that I just experienced?" If so, then the system is behaving normally. If not, investigate.
3. Ask myself, "Have I experienced a 25% drawdown yet?" This is my hard limit for a drawdown. I will stop trading, lower the position sizes, and start over if this ever happens.

If I follow this, then I should be a consistent operator of a mechanical system.

Click to expand...

As a builder of mechanical trading systems, I must say that I am impressed at the level of objectivity here. Only one last hurdle to overcome: the 'hard drawdown'. It is good that you have a hard stop so to speak, but make sure that your strategy is setup so that only in extreme market conditions that this hard stop would be hit. That 25% should not be arbitrary; as your strategy must be allowed the space to be profitable the majority of the time.

And forward testing would be wonderful. Most brokers will front you a free demo so you can get a good idea of how your strategy performs before you go live. Are you using any form of automation?

I've also built many automated trading systems and found that the two most important points are diversity and testing.

A strategy can always fail because the market changes, and can fail so bad that not even parameter re-optimization can save it. It's good when you trade with not one but many strategies and in many different markets.

And good testing is important, not only the strategy should be tested, but also its optimization process. There's a method called Walk Forward Optimization that is very useful for getting some reliable predictions how the strategy will behave in the real market.

What about the performance evaluation? Which parameters matter? Which ones are junk?
I'm referring to win rate, avg trade return, profit factor, etc...

natureboy said:

What about the performance evaluation? Which parameters matter? Which ones are junk?
I'm referring to win rate, avg trade return, profit factor, etc...

Click to expand...

I think the win rate is meaningless. You can write a system in five minutes with a 99% win rate and that system is still losing.

In my opinion the profit factor, the Sharpe ratio, and the profit to drawdown ratio are good performance parameters.

Of which the profit to drawdown ratio can be misleasing because the direct drawdown depends on the order of trade outcomes, which can be random. When the drawdown is determined indirectly f.i. by a Monte Carlo method, then the profit to drawdown ratio is a very good performance parameter.

wheresthejelly said:

2. Design a system that meets his/her performance goals over a specific type of market (e.g. the US bear market from 2000-2003). Then, the designer should forward-test into the next few years (2004-2007) with parameters (i.e. MACD bars, RSI threshold, stop loss amount, etc) that were successful from 2000-2003. Then, optimize the parameters across the time period 2003-2007, and use these values for the next time period from 2008-2011. Once the forward simulation reaches the end, the designer looks at the equity curve and drawdown to see if it meets the performance goals. If not, tweak the system.

Click to expand...

Sounds like curve-fitting 2 me. soz buddy.

GTTY.

natureboy said:

What about the performance evaluation? Which parameters matter? Which ones are junk?
I'm referring to win rate, avg trade return, profit factor, etc...

Click to expand...

Just more distractions. None of those really matter. The primary performance evaluation is seeing if you get a net account increase over many repetitions or tradecycles. Then repeat these tradecycles as much as possible.

I don't seem to be allowed to start new threads; this thread seems relevant though so I will post here.

Contrary to what my post count may indicate, I am a longtime lurker of this and other trading forums. I am not looking to sell any wares, I am not looking to have an esoteric discussion about "trading psychology", and I am not here to profess the wonders of "price action" trading.

If it helps my credibility, I am familiar with posters from some time ago such as Bramble, EliteJets, Trader_Dante etc. This is not me vouching for any of them and nor do I have anything to do with such posters, I am just trying to show that I am not a blow-in.

I am someone who has learnt how to trade discretionally, and am ready to move into the realm of mechanical trading. I would like to seek some advice, if anyone would be willing to share. Trolls need not apply (but will anyway).

With reference to this thread, my question comes down to what constitutes a successfully backtested mechanical system? I suppose what I would like to know is "How much is enough" with regards to testing? I realise that there is no definitive answer to this, so let me tell you some of what I have done and maybe you can add to my thoughts.

I have developed a mechanical system for equities and futures. It work across multiple equities (not all), multiple timeframes (incl. intraday), and multiple exchanges. My success across these different realms leads me to believe that I have a bonifide edge. My results are not marginal.

The backtesting has been completed mechanically, but not in your typical manner. I know the many limitations of EasyLanguage and NinjaTrader, and prefer to have the confidence that can only come through manual backtesting. Yes, testing in this manner takes time.

Yes I have forward tested over several months. Due to the nature of my backtesting, the results from my forward testing essentially match the results from backtesting.

Yes I have completed Monte Carlo testing of my results (although I question the usefulness of the results, due to actual trade performance being unique).

The system has not been optimised in any way and I do not intend to optimise for fear of curve fitting. If this means that I underperform in some manner, then so be it. Frankly, the results are more than suitable as they are.

So back to my question, how much is enough?

The system generates ~2.7 trades per day. This means that on any one instrument (equity/future), I will generate 800-900 trades per year. Obviously more is better for statistical purposes, but there are obstacles to testing, particularly with regard to data quality.

Yes, quality data exists, but I have found it to generally be prohibitvely expensive at this stage. Hence, any data obtained through other means usually needs to be quality controlled for missing/erroneous data. This is quite the exercise if you are using say, 5 minute charts.

At what point do I say, "be done with it, time to go live?".

You are always at the mercy of the market, no matter how much testing you have done.

jaella said:

.......

At what point do I say, "be done with it, time to go live?".

You are always at the mercy of the market, no matter how much testing you have done.

Click to expand...

When you are confident in the forward tested results performing as expecting in typical market conditions.

There is 1/2 of 1/2 of systems implementation that we have control over and 1/2 that we have no control over. Focus on the 1/2 that you have control over; mainly these are entry, position sizing, and exits.

No amount of testing, forwards, backwards, sideways will be enough because when you want to take someone's money for real they will fight you every part of the way. But you already know that since you submit to the market's mercy. The market ain't dumb like your simulation.

By manual backtesting, I mean going through the historical data by hand, as opposed to programmatically backtesting. I am forward testing with each new day that passes and am into my second month of forward testing. Just getting a little antsy, because I know that I am onto something.

ybfjax, that was what I was thinking. Perhaps I will hold out until the new year.