Backtesting puzzle

[rsmith911]

I was getting some strange results with MT4 backtesting so I created a simple program that just places orders with equal stops and take profit limits. I have several years of 1 minute data downloaded so I ran the program over that in strategy tester. The program creates buy limit orders and I also ran the same program with manual stops/profits and also not placing any orders just tracking the price and incrementing/decrementing a counter when the stop/profit are hit. What I found with all of them is that with a small value for the stop/profit I get about 65-70% of orders hitting the stop. At large values for the stop/profit I get more like 50% which is what I was expecting since the stock market is supposed to be roughly a random walk.

Can anyone explain why this would be? In my mind the stop/profit value shouldn't make any difference if they are equal- there will always be fluctuations in price at whatever period you look at the prices. But these should be both up and down and so you should be equally likely to flucuate up and hit a profit as you are to fluctuate downwards and hit a stop. Thanks!

 

[the hare]

Are you including the spread in these tests, it has a massive effect on win rate when using small stops & targets.

In the past, I've done similar tests on random entries, and I got pretty much the same result as you did

 

[rsmith911]

Are you including the spread in these tests, it has a massive effect on win rate when using small stops & targets.

In the past, I've done similar tests on random entries, and I got pretty much the same result as you did

Well since they are limit orders I don't pay the spread.

I wonder if this is the reason - for the buy limit order to get hit the price has to fall by 1 point from the current bid. So if the price is already falling maybe that predisposes it to making it more likely to continue to fall and hit my stop first if the stop is small (although I would have thought that isn't very random walk like). When I widen the stop/profit then the order has time to ride out any initial downwards tendancy and the true 50% behaviour comes out.

 

[Joey25]

Well since they are limit orders I don't pay the spread.

I wonder if this is the reason - for the buy limit order to get hit the price has to fall by 1 point from the current bid. So if the price is already falling maybe that predisposes it to making it more likely to continue to fall and hit my stop first if the stop is small (although I would have thought that isn't very random walk like). When I widen the stop/profit then the order has time to ride out any initial downwards tendancy and the true 50% behaviour comes out.

Chances are those orders are really market-if-touched orders, meaning that you still pay the spread, and hence perform worse for smaller bracket orders.

 

[the hare]

I wonder if this is the reason - for the buy limit order to get hit the price has to fall by 1 point from the current bid.

IIRC the people at Automated Trading Systems and Robots developed a random robot that took random entries, but determined trade direction on the basis of the last tick. I cant recall what they used as an exit, and I havnt done a great deal of testing with tick data, so I wouldnt like to comment.

And as pointed out above, if you use MT4 with a retail forex broker, you will pay the spread, so in reality the losers have less distance to travel than the winners, hence the higher frequency of losers. Of course you might not have chosen to include the spread in your simulation, in which case you'd expect something closer to a 50/50 distribution. If you've choen to ignore spread, is the expectancy equal ? or is the average win increasing to offset the lower win rate ?

Accounting for the spread, I've generally found the expectancy to be roughly equal and independant of the size of stops and targets (although I'll admit I've not done anything like enough testing on this)

 

[rsmith911]

I am using an ECN demo account and for limit orders so I don't pay any spread on a buy limt - take profit order (just the commission which is charged later). However on the stop loss there is slippage of course.

However, I am only interested in the number of orders that hit either the stop or the limit - not the profit/loss that results thereafter. If I scroll down the list I can see there are virtually no TP results in a sea of SL results. TP & SL are both executed on bid price so they should both be triggered at an equidistant point from the initial order price. Therefore all things being equal I'd expect equal numbers of TP & SL results. This isn't what I see however - the smaller the TP/SL distance the greater the % of SL results. The only thing I can think of is the fact that the buy limit order is triggered by a downward movement to start with and therefore I'm always buying in a down trend of sorts?

 

[rsmith911]

Well I just simplified things by altering my program to create market orders, then get the price of the order opened and set the TP/SL and equal distance either side of it. Then I wait for the order to close and increment/decrement a counter depending on whether the order closed at a higher or lower price than it opened. So profit/loss, spread, entry price etc. don't come into it. All that matters is price fluctuation after order opening.

I get the same results. If I set the distance at 15 points I get 25% TPs. If I set it at 150 I get nearly 50%. If I reverse the orders and place sell market orders instead I get the same pattern - more losses than profits with small stops.

Don't get it

 

[Shakone]

Sorry, I don't use MT4 backtester but I don't understand how you can say the spread doesn't come into it. If I enter at 100 (with a limit order or market order, the bid ask will usually be 99/100) with stop size 15, and target size 15. Then my stop is at 85, my target is at 115. In order to get filled on the target, really I need price to move to bid ask 115/116 (a 16 point improvement from 99/100), yet to be stopped out, it only needs to go to 85/86 (a 14 point move against me). Therefore it's more likely to go 14 points against than 16 points for me.

Now if I double the spread to 2 points, we'd go from 98/100 to be filled initially at 100, and have target at 115/117 (17 point move in my favour), stop at 85/87 (a 13 point move against). If you have stops and targets much alrge than 15, then of course this spread would have less of an effect.

 

[rsmith911]

Thanks, yes that explains it for the market orders.
But if I go back to my original code and place limit orders then the entry price = the bid price so if I close my order immediately I make neither a loss or a profit.
So in this case the spread should have no effect right? If I place my stop/profit at 10 points that means the bid needs to fall or rise 10 points from the entry price to hit either. But then I guess I am back to square one in that I don't know how much effect the downwards movement of the price leading up to entry has...

 

[Joey25]

Thanks, yes that explains it for the market orders.
But if I go back to my original code and place limit orders then the entry price = the bid price so if I close my order immediately I make neither a loss or a profit.
So in this case the spread should have no effect right? If I place my stop/profit at 10 points that means the bid needs to fall or rise 10 points from the entry price to hit either. But then I guess I am back to square one in that I don't know how much effect the downwards movement of the price leading up to entry has...

They're still market-if-touched in the sense that (for a buy order) the ask needs to touch your price. In other words, the bid is below your price by the amount of the spread. A true limit order earns half the spread, because you only need the bid to hit your price.

 

[Shakone]

Thanks, yes that explains it for the market orders.
But if I go back to my original code and place limit orders then the entry price = the bid price so if I close my order immediately I make neither a loss or a profit.
So in this case the spread should have no effect right? If I place my stop/profit at 10 points that means the bid needs to fall or rise 10 points from the entry price to hit either. But then I guess I am back to square one in that I don't know how much effect the downwards movement of the price leading up to entry has...

As Joey's explained, your assumption about limit order fills is not correct.

If I place a limit order at 10 and the bid ask is at 12/13. Just because the bid ask gets down to 10/11, it does NOT mean that my limit order was filled at 10. The limit order sits on 10, until someone is willing to take that price. It might be filled, it might not. For backtesting purposes, you can assume it wasn't filled and that the bid ask needs to go lower to ensure a fill.

 

[rsmith911]

Brilliant thanks guys, I used F12 to step along in the visualizer and I see you are right and my orders aren't being opened until the ask hits my order price. So as you say the bid is already below my opening price and therefore closer to the stop :clap:

Sorry if I am being a bit dim but can you just confirm a couple of things for me:

In a live situation presumably my order *could* be hit before the ask reaches it (if someone places a market sell order and I have the best bid) it in which case it would open at that bid price and I start with no disadvantage - 50% likelyhood approx. of either stop or loss being hit.

I'm not sure I understand what you mean by half the spread. If someone places a market order that is matched against a limit order the difference is 1 spread isn't it? So where does that money go? Are you saying the person with the limit order only gets half of that? If so who gets the rest?

So it seems to test out the behaviour of limit orders properly I need to run off a live account and see how many (if any) get filled before the ask price hits them. Incidentally if I were to place a buy and sell limit orders inside the spread at the same price would the system match both of my orders together and close them both off with each other?

Cheers!

 

[Joey25]

My understanding of the MT4 platform is that it is not a limit order book. Traders always "cross the spread", so passive limit orders are not possible. Therefore, your order is only triggered, as you have said, when crossing the spread will result in that price. This is a market-if-touched order, so you could get a different price if markets are fast, because your buy order will pay whatever the ask is whenever the order is transmitted.

With regard to true limit order books, if the price quoted is say 99-100, then a passive trader has to have a bid order AND an ask order filled by an "initiating" trader to earn the full 1-point spread. It is in this sense that I referred to a single limit order as earning half the spread.