The Tenets of a Successful, Backtested, Mechanical Trading System

[Shakone]

Going through by hand is useful, but is also dangerous, as we have hidden biases, and look for what we want to see, and ignore some things we don't want to see. So it's easy to get misleading results from this.

 

[jaella]

Going through by hand is useful, but is also dangerous, as we have hidden biases, and look for what we want to see, and ignore some things we don't want to see. So it's easy to get misleading results from this.

Yes, I understand what is involved. I have removed the possibility of bias in my backtesting. Excel is used to an extent. I was trying to make the point that my results are not due to some programming quirk. I find testing in this manner to be very transparent, hence why it performs similar when forward tested.

 

[jaella]

Is there any reason why you think simulating trades manually is better than letting a computer do it?

Aside from the mentioned problems of unrealibility, I imagine going through years of charts and simulating trades is also extremely boring . Any significant tests, such as walk forward analysis, can not be done manually anyway. You can do only simple backtests with fixed parameters and they do not tell much about the performance in real trading.

When backtesting programmatically you have the annoyance of intrabar granularity and how the program deals with this problem. A solution can usually be coded, but it complicates even the simplest of backtests. This is not generally a problem if your entry/exit is onbarclose, but if you are using a threshold that can occur within a certain time period then you will have issues.

The parameters that I use are dynamic and adjust with the market, allowing for simple testing.

As for being boring, I can handle the monotony because I know what this is potentially leading to.

 

[Liquid validity]

When backtesting programmatically you have the annoyance of intrabar granularity and how the program deals with this problem. A solution can usually be coded, but it complicates even the simplest of backtests. This is not generally a problem if your entry/exit is onbarclose, but if you are using a threshold that can occur within a certain time period then you will have issues.

I agree, Intrabar granularity is one of the biggest stumbling blocks anyone will face in
backtesting / automation, if using time series data.

Many discretionary rule based methods (which otherwise work) fall apart at the seams for this very reason
when backtested / fully automated.

I don't know about other platforms, but Ninja can for instance, reference
daily charts (or any other tf for that matter) whilst executing on a lower time
series, or even tick level.

 

[Liquid validity]

By manual backtesting, I mean going through the historical data by hand, as opposed to programmatically backtesting. I am forward testing with each new day that passes and am into my second month of forward testing. Just getting a little antsy, because I know that I am onto something.

ybfjax, that was what I was thinking. Perhaps I will hold out until the new year.

Personally, 2 months wouldn't be enough for me.
I would forwards test until:

1/ First major drawdown reached.
2/ 300+ forward test trades.
3/ 12 months forwards test duration

Once all 3 criteria fulfilled, forwards testing ends.
Not hard and fast rules, just my own preference.

 

[wheresthejelly]

Just wanted to update everyone here and thank some of you specifically for things I've learned while reading this thread since I made the original post.

I started trading live my completely automated mechanical system in late October 2012. I am now up 12% since then with a 5% drawdown so far. These numbers are nice, but here is what really made me happy: my live performance closely tracks my simulated performance over that same time period.

Thanks bbmac for getting the replies started in a positive light.

Thanks Spec-K Trading for exploring other angles of mechanical systems. Too many points in your huge post to reply to everything right now. I pretty much agree with all the basic ideas behind your post. My system trades many different markets with lots of independent trades every day. My overrides are in place in case of a black swan event. My system backtests about 20,000 trades between 1990 and 2013 with just over +1% trade expectancy. If I open my pre-filtering a bit, I can get 60,000 trades with about +0.5% trade expectancy.

Thanks doh for pointing out forward testing. My system is very adaptable to changing market conditions and doesn't use many fixed parameters. Forward-testing did not prove useful for me. Optimization over the long time frame of 1990 to 2012 proved VERY useful.

 

[Spec-K Trading]

Congrats on your live trading success Jelly. I've also had great success now for 10 months running, trading several hundred thousand continuously (198 trades completed to date). +22% currently, and only one losing month (Feb 2013), with new equity highs again this month (Mar 2013). And yes, overall I'm running at about 1.1% average per trade return. My system keeps all the money active all the time, spread across 10 trades. The faster I can reenter on a profit exit (which on average occur several times a week, 2-5 typically), the faster returns grow, so I frequently am reentering same day as exit.

I'm more than a little aware the market has been sweet for the most part for long-only systems, and I'm sure to have a significant drawdown when the current market cracks, but that's just part of the game, I intend to trade through it (because that's how I found I maximize overall profits for the long term, vs. back off strategies that I found always reduce long term profits with my system).

As for the many questions about when enough testing is enough, there is no right answer. What I'd suggest is when you feel confident based on all the testing, start putting your money to work, ideally incrementally (scale up). That's what I did, starting with 1/2 size trades, and scaling up over the first two months as I got comfortable with serious $'s in the market all the time, etc.

 

[numbertea]

Complete automation of a profitable trading system, also staying profitable, is a dream that I have been cooking for the past 5 years. Backtesting and forward testing is of course the only way to test. I have not found a commercially available demo that is completely realistic so I had to build one into my own system so that I could test it. The importance of granularity on intrabar trades wasn't rocket science to solve, level 2 book data and trade que tracking was the only way that I could finally reproduce actual market conditions. I use Lightspeed API and with a fast Internet connection I've been able to have results of testing meet results of real world trading. Now if I could only quit my job and spend more time programming I could quit my job. Do I have to really keep paying my bills?

Cheers all

 

[graydrake]

I find a lot of posts in this forum are asking questions rather than providing solutions. So, I'm going to provide what I believe to be the major important tenets of a mechanical trading system. I see a lot of confusion out there about what makes a mechanical trading system successful. This post should clarify what a system designer should aim for and what he/she should believe about trading.

Major tenets and beliefs:
1. Human traders or discretionary traders lose trades because of fear and greed.
2. Human traders or discretionary traders cannot maintain the necessary discipline to trade consistently forever.
3. Market movements of all kinds (price, volume, volatility) are 99.99% determined by human factors rather than mechanical systems.
4. Every profitable trade in history will be repeated in the future.

These are beliefs, which I feel are very close to reality because I have tried discretionary trading, and I also design mechanical systems. I have been lurking on T2W for about a year, and I've designed two mechanical trading systems. Full disclosure: I have not made any real money with these systems yet, but I feel that I will in the next few months once I move away from paper trading. I am not fully qualified to tell you what works in a real-money system. However, I am qualified to tell this forum how to design a system that backtests very successfully.

Just starting out last year with backtesting software, I could barely get an upward sloping equity curve in simulation. Now, I've got an upward sloping equity curve with very low drawdown over a span of 20 years. Some simple stats I can give you are my backtested cummulative annual return (53%), and the maximum system drawdown (28%). Both of these stats are over the time period from January 1, 1992 to June 29, 2012. I've learned so much in how to design a successful system in backtesting software over the past year. So, I am qualified to tell this forum how to design a successful system in backtesting software.

Key Success Factors in designing a successful mechanical trading system in backtesting software:
1. Performance goals
2. Accurate and comprehensive data
3. Positive expectation
4. Risk management
5. Time frame
6. Optimization

If you can understand and do well at each of these areas, you will design a profitable mechanical trading system in simulation. Here's more elaboration on each of these key success factors:

1. Performance goals: Decide for yourself BEFORE backtesting anything what drawdown you can handle. Imagine you had a working, real-money system. Then imagine that next week your system lost 25% of your money. Would you break down and cry? Would you stop trading out of fear? If you feel a cringe at losing that much money, try 20% or 15%. If you're okay with 25%, what about 30% or 35%? Another stat you should define is time in a drawdown. Let's say you recently had a good month of trading, and your account value is the highest it has every been. Feeling good! Then, let's say your system loses 10% next week and doesn't recover for 6 months. Could you handle that? If you trade for clients, could your clients handle that? What about 5 months? The last stat to define is your desired cumulative annual return. Would you be happy with 25% per year? If so, what about 20%? Keep going down until you find a number that sounds unacceptable to you. These three statistics are your performance goals. Design your system to achieve them. Change your system if you feel it doesn't meet these goals. Be serious and realistic about this! Don't be silly and choose a 5% drawdown, 2-week time in drawdown, and a 300% cumulative annual return.

2. Accurate and comprehensive data: Your data source should provide data for all the stocks you want to trade. Also, it must provide data for all de-listed stocks you would have traded. Make sure you go through some stocks to make sure that stock splits were handled correctly. Some backtesting software and data sources do not account for dividends. It is up to you to decide whether that invalidates your backtest results or not. A system that has an average trade length of a few days should be fine with not accounting for dividends. A system that holds positions for weeks or months should somehow account for dividends. Remember that short positions pay out dividends rather than receive them.

3. Positive expectation: It is very very important that your trading rules have a positive expected value. Mathematically, this means (win rate)*(win amount percentage) - (loss rate)*(loss amount percentage) > 0. For example, let's say you design a fictitious system that buys on a moving average crossover and sells 4 days later. Your system wins 58% of the time and loses 42% of the time. Your winning trades make 3.5% on average and your losing trades lose 3.7% on average. Your expectation is 0.58*0.035 - 0.42*0.037 = 0.00476, which is greater than zero. This means that you can expect to make 0.476% on average on every trade you enter in the future (Tenet 4 above). It is very difficult to design a profitable trading system with a negative expectation.

4. Risk Management: Risk is the potential that a particular trade you enter will results in a loss. Every trade you enter has a chance of losing. Some trades have a bigger potential loss than others. You can find out some factors that produce bigger winners and smaller losers. Perhaps stocks with a very high average true range win big with your system? If so, bet more on those positions and less on others. Also, consider how big your stop losses and profit targets are if you use them. There are many position sizing methods to use here, but the main ones are "fixed risk" and "fixed fractional" position sizing. The Kelly formula will give you a decent starting point for position sizing. Google that.

5. Time frame: Just about every random mechanical system ever created is very successful over a short time frame like a month or a year. However, you have to know when to start trading that system and when to stop. A short-only, trend-following system would have been great during the time frame of September 2008 to March 2009! How do you know when to stop trading the system? It's very hard to predict turning points in the market like March 9, 2009. It's much simpler if you design a system that performs well or at least okay during up, down, volatile, and quiet markets. It's okay if you can get your system to make hordes of money in some types of markets and just limit losses other types of markets. The key statistic is your expectation over a long period of time (like 10-20 years). Hopefully, it is positive. If you have positive expectation, you can adjust your risk management to achieve your performance goals.

6. Optimization: Run your backtest over your selected time frame with a range of different parameters. For example, going back to our moving average crossover system, test over different periods for the moving average. You can test a range of values for the periods of any indicator, the stop loss amount, the fixed risk amount, the number of days to stay in a position, etc. Any parameter that is numeric can be optimized. In your optimization, you want to make sure that your system makes money over a wide range of values for each of these parameters so that your simulation results are robust. Once you confirm that, choose the parameter values that produce the best performance (e.g. highest annual return, lowest drawdown, highest win rate, etc.).

I hope this helps some people and stirs some discussion. Thanks for reading!

Is there anyone out there that has successfully used a truely mechanical system?

If so,
What was your annualized return?
Over how long a calendar time?
How many trades completed?
Win/loss ratio?
What market?

I have successfully back tested several. Unfortunately they work in todays back test because the shape of the index (or ticker) price/time movement is compliant with success in the specifics of the mechanical system. Tomorrow's shape will be unique and almost certainly will require an alternate mechanical system to deliver similar results as in the past.

Comments?

Drake

 

[robocop]

Here my contribution of how I screen the results for a backtest without having years of intraday tick data available.

The first thing I look for is a nice and steady equity curve, like this system on the DAX with 5 min bars.



And has good figures.



Then I test it for robustness by running an optimization report with a different set of combinations. If almost all combinations have a profit factor above 1 then I can look further to see which combinations give the best result.

I also look at the best combinations and the worse combinations to see similarities. It is possible that the system doesn't show good results at the start. Playing with the timeframe and atr filter can help a lot to optimize the results. The atr filter helps to filter out any noise.



The robustness can be tested in many ways. If it works on 5 min bars, it should work on 15 min bars, but you will need more money to trade the system because your stoploss is more far away.

In the chart below I only have changed the period from 5 min to 15 min bars. Everything else equal. Notice that is still profitable (profit factor of 1.43).



Now I can run optimization for 15 min bars to optimize it further.


The most optimized combination gives the following equity curve with a profit factor of 2.27.


This is with only an 300 euro account. What if I had a 3000 euro account. Does the system change character? It doesn't matter so much. I now have a profit factor of 2.05 and it still shows the same system characteristics.



There are lots of other tests that you can do, but this is the way I screen symbols.

If you have any questions :?: or suggestions :idea:, please let me know.

 

[robocop]

The test starts at February 4th 2013 and uses tick data.

View attachment StrategyTester.htm



Two months ago it gave me more or less the same equity curve and profit factor with the data I had then.

I want to backtested it with more data, but I don't have any. Where do you guys get such data?

 

[robocop]

I've run the test with open prices only and with tick data prices. The backtest give exactly the same results. So I guess it doesn't matter so much for my system.

The system takes both long and short trades, but the prices of the broker are only bid prices. Therefore I get stopped out more often with short trades, due the spread. Do you have the same issue and how do you cope with it?

Can Metatrader 4 calculate the Sharp ratio for me? I don't see it on the strategy report.