Follow the development of a new trading system

Hi,

we've been developing trading systems for a few years, and recently decided to let traders follow our process where we explain all the steps we go through to develop a new system. This has previously only been avalible in swedish, but since we've decided to write in english instead we want to share this with everyone. The two first parts will you unfortunately miss out on, but the third part and the upcoming articles will be published here in this thread.

Please do not fill this thread with unnecessary comments, let us instead try to keep it readable and open for discussion regarding the techniques and findings of the research. The following is an excerpt from the website. I'm not sure I'm allowed to post link here, but for a better view of the article you might want to try to find the website.
----------------------------------

SYSTEM DEVELOPMENT PART 3 - Multidimensional distribution

After more in-depth analysis of the previous research experiment, we decided to change the idea completely due to a low opportunity factor. Even If we had a positive expected return, it involved a pretty large risk and the trades weren’t as frequent as we desired. Therefore, we’ve decided to go in a new direction with a more statistical based approach.

The system is based on the previous requirements, such as:

- The system shall be fully automated
- The system shall be suited for the forex markets
- The system shall work on different timeframes, hence increase the opportunity factor
- The system shall give us 2/3 winning trades with equally large average profits and losses, or 50% winning trades where the profits are double the size of the losses

The main idea is that forex markets, where two currencies are linked to each other, most of the time fluctuates around a certain “balance point”. We’ve chosen to call this point “equilibrium”, which will show the average price of the two currencies in relation to each other, for a certain time. The forex markets usually trend a lot, but the circumstances doesn’t allow the price to increase to a theoretical infinity (as in the stock market), or decrease to non existence.

So, we’re taking a step back with the purpose to define if the market will give us any opportunity for this kind of trades. The first thing we would like to see is if our idea will give enough scope for the price to move after our entry is taken. For this, we’ve created a multidimensional distribution analysis, to see how the market move after the entry is taken. This will eventually end in a probability model for different scenarios, where our stop loss, profit target and money management will be based on the current market behavior and probability for each scenario.



Graph one shows the perspective view of the three dimensional distributions. The distribution is made up from the closing distance from the equilibrium (our entry point) on the horizontal axis, and for each bin for these values the deviation from this equilibrium on the depth axis. As we can see from the chart, most of the trades where gathered around the equilibrium. The two minor peaks around the large peak symbolize trades that started trending after entry. This is actually undesired in this case, since we ultimately want the price to get back to the equilibrium. The two tops makes up an equal of the height of the center top, which means that as often as the price found its way back to the equilibrium, it also found its way away from it.



However, if we continue to look at graph two which shows the same multidimensional distribution from the top view, we get a different story. The distribution is split by the centerline, which symbolize the balance between deviations of increase in price and deviations of decrease in price. As we can see the distance (deviation) from the extreme values (highest and lowest) during the time in the trade to the close is much greater, than the deviation from the close to the equilibrium. This tells us that the fluctuations towards extreme values are more likely to get back towards, but not completely, towards the equilibrium.

Now we have one positive behavior, the fluctuations where the price searches itself toward the equilibrium, and one negative, where the market trends as often as it gets back to the equilibrium. We have already got a hint about how money management might improve this strategy. But, we can’t make further conclusions without going more in-depth with the statistics at this point and with further testing, so this is where we bring the analysis to an end right now. The upcoming report will deal with the next step, either the probability for different scenarios or further analysis of the market opportunity.

Best regards,
Johan Andreasson, System Investors

Now we are interested in how the price moves in relation to the random normal distribution, and also how this changes over different time frames. Based on the previous analysis, we can make the conclusion that when prices keep moving inside a smaller number of deviations; it (the bar) tends to close near the equilibrium. Furthermore, when price moves away a certain amount of deviations from the equilibrium (if we compare it to the normal distribution, the equilibrium is the mean), the bar tend to close further away which indicates a trending scenario.

So, what does this mean? Basically it means that either the market moves sideways in a relatively narrow range most of the times, and when it breaks away from this range prices tend to move far away from the equilibrium. The probability for the price to move back to the equilibrium is still higher than the probability to move away, but compared with the random normal distribution this tendency has a higher probability than when random prices is used. In practice, this means that the trader can benefit from two kind of strategies – swingtrading/scalping will work great when the market is trading within a range, and trend following strategies when the market moves outside a certain number of deviations (based on the charts, the threshold is approximately 2.8 deviations).







As you can see in the graphs below, the 5m timeframe shows the highest tendency to act as a random normal distribution, while the H4 timeframe instead tend to either retrace to the equilibrium or trend far away. Please note that in this analysis we have not considered the time effect, it is only based on volatility. It is not impossible that the price tend to retrace to the equilibrium more often during low volatility times (such as during the night session) and to trend during the higher volatility times.





Nothing new, right? Perhaps not, but this analysis provide us with an interest idea for both positions sizing (which we will run a simulation on next) and also how to implement different kind of trading tactics based on the market scenarios. First, I want to explain some why: I assume most readers are familiar with the concept of normal distributions and standard deviation. Consider a simple strategy that is based solely on price, where we assume that the market move in a completely random fashion. This would mean that the probability of a certain move could be predicted with help of the normal distribution. The probability that the price would reach +1 standard deviation during the measured time unit is therefore 34.1%, and for +2 standard deviations is 13.6%. We will now look at the strategy where we go long at the beginning of the measure period, take a profit once +1 stdev is hit or a loss once the -2 stdev is hit. If this strategy was built on a bet, that we received one $1 dollar and risked $2 and if no level was hit the bet was cancelled, we would have a positive expectancy (0.341*1-0.136*2 = $0.69). Unfortunately, this is not how it works. Since the random normal distribution don’t have binary values, but instead is made up of a probability density function, this means that once our chosen period of time has ended we will end up with any value between $1 in profit and $2 in a loss, if none of the levels is reached. If we increase the levels in the previous test to binary steps each 1 stdev, we would instead have the expectancy of $-0.136 (0.341*1-0.341*1-0.136*1) since it is equally high probability that we end up with $1 as -$1, but then we also have the additional risk that the -$1 turns into a -$2. By running a monte carlo simulation of 10000 iteration to introduce non-binary steps we can see that this gives us an negative expectancy. This is because we limit our profit potential at the same time we have a higher potential to lose twice as much.

Now, how does this apply to the market? Since the market doesn’t move exactly as a random normal distribution as you can see in the graphs, we can’t apply this piece of theory direct to the market. In order to see how the same kind of strategy will work on the forex market, we need to try it with a probability density function that is similar to the markets distribution instead. However, this will take a lot of work and analysis to find such a probability density function, therefore we will not dig into that problem now and instead end this little article for now.

Best regards,
Johan Andréasson, System Investors

Our next step will be to implement the knowledge from the previous research. As a short summary, the Forex markets tend to either go back to the equilibrium or move far away from it. Compared with a Random Normal Distribution it’s less likely that the price moves (and stays) somewhere in between, hence behave similar to the normal distribution. Based on this, we ultimately want to trade the counter moves when the likelihood of the price returning to the equilibrium is high, and instead follow the trend when the probability is higher for continuing trends. The problem is then – how do we do that? And how do we determine if price will move away or fluctuate close to our equilibrium? How do we even determine what price the equilibrium is?

One step at a time. We will start at a very basic level, to make sure our assumptions and ideas are not completely wrong. We will start with pretty common oscillating indicator. Unfortunately we won’t give away all details throughout these articles, as they hopefully will result in a useful system. However, consider a common indicator that behaves as an oscillator, such as stochastic or RSI. These are based on some sort of equilibrium, since they indicate oversold and overbought levels. The major problems with these is then what conditions to use, since if the market reverses too quickly and we enter when the indicator turns down – we might get in too late and the move is over. If we enter once the indicator reaches its extreme levels – we might get caught in a trending market while holding an opposite order of the trend.

The following graphs show a simple strategy based on simple conditions for the oscillator indicator. Please note that we usually do not use Metatrader’s standard graphs from the back tester, since we think they give the wrong impression since it does not consider the time variable. Instead the equity curve is exported at a certain frequency (such as every 4 hour) to better determine how the trades affect the equity growth. These graphs therefore show the equity curve, where the first one is based on entries when the indicator reverse from extreme levels, while the second one opens an order immediately when an extreme level is reached. The different shades in the graph show the performance in different pairs.





From analyzing the equity curve and the actual trades on the charts, we can draw a few conclusions. Among these is that during high noise the strategy which enter when extreme levels are hit works better, but this one is easily caught in a trend with a losing position. On the other hand, the strategy with entries when extreme levels are reversed, will perform better in these trending markets since we stay in the position when the trend is initiated, and exit (reverse our position) when the trend show tendencies to end. The downside is at the same that in noisy markets we will get it too late and end up being whipsawed.

The “extreme reverse” strategy often got caught with an opposite position in trends since we got an entry signal, but the market quickly reversed without giving an opposite entry signal, thus leaving us with the wrong position. Therefore we did a simple test where we exited our position if we were about to get caught in the opposite direction, but this didn’t help much. The average loss was lowered with about 30%, but the number of orders increase and led to more whipsaws and more losses due to spreads.



These tests confirm our beliefs with the problems previously mentioned – during noisy markets we want to quickly enter and exit a trade on extreme levels, and when the market tends to trend or show more swing movements, we prefer to stay in longer since the indicator will stay on extreme levels during the trend. To gain more understanding in this, we ran some additional tests with different input values. The following charts show different “sensitivity” values as well as different thresholds for extreme levels. All values showed a pretty boring result during the first few years (2005-2008), and then crash and burn when the volatility suddenly increased in the market. During the calm years an average sensitive value showed the best result, while the two more extreme sensitivy levels (high/low) had lower drawdown during the last year. This further confirms the logic with noise/trending markets.



Before we move on to trying to develop a solution that take advantage of the problems so far, we want to try another technique that make the actual sensitivity value of less importance, since the performance so far is highly dependent on the sensitivity variable. This technique might be one of the key factors to the strategy’s success; therefore we will not reveal what it is (at least not yet, sorry). The graph below shows the result when this strategy what used. The benchmark is EURUSD with the same basic settings as the new technique. As you can see, the new technique gained an advantage during the first few years, but crashed when the market got volatile. Basically, the new method has increased the importance of finding a solution to the noise/trend problem, which we will focus on in the next article. If we can find a good solution, the possibilities with the new technique can lead to significant gains.



Best Regards,
Johan Andréasson, System Investors

System development part 6

Now we’re back again after some time. The development has continued and we’re moving in the right direction. To relate to the previous posts, where we talked about developing a filter that determines the ratio between noise/trend in the market, we have more or less put that idea aside. The indicator we created had some advantages, but in relation to the strategy that we try to develop now and based on the basic technique for the strategy, we never got any synergy effect. When the indicator signaled that we had a trending market, and our strategy tried to adapt to that, it was too late and the market instead started to fluctuate. Other strategies will benefit from this indicator, so we won’t throw it away, but for this current development it will no longer be considered.

So, what’s new then? Well, some of the ideas from the discarded trending filter are still interesting, so instead we incorporated them directly into the entry strategy (the oscillator technique). We succeeded to find a certain relationship with this technique, the time periods and the market behavior. For all of you that develop your own systems, I would like to pin out that a common “mistake” many developers make is that they don’t consider the relative change. For instance, one who tries to use multiple moving averages in a strategy might set them to 10-20-30-40-… et cetera with 10 period steps. This might seem reasonable, but the larger the period gets the smaller the actual effect will be. Don’t get me wrong, I’m not saying this is incorrect as strategies might benefit from more focus towards the higher periods. However, in case the developer tries to spread the effect, s/he shall consider the ratio instead. A sequence of 5-10-20-40 would give more equal weighting to faster (shorter period) moving averages as well as slower (longer period) moving averages. Some people prefer to use Fibonacci series, even though I personally don’t believe in their use in the current context.

The results from this adapted entry technique is better than the one we previously got. Please note that we still have no filters, money management or other parts that will affect the result. Our goal here is to get a breakeven strategy, as later stages will look at profit opportunities to make sure the spread won’t eat up too much of our profit. If we can get a strategy to breakeven with thousands of orders, this means that we made a profit equal to the spread on average. The following shows the result of a few random pairs on the higher timeframes, as the lower timeframes did not provide enough profit opportunity (the range was too small compared with the number of trades and size of spread), which resulted in a slightly downward sloping equity curve. On longer timeframes however, the result varied form “decent” to “good”. The following two graphs show 3 pairs from the 4H and D timeframe, where the red line is the average of all three:




What we will do next, can be somewhat complex in practice even if the theory is more logic – we shall examine the profit potential before the order is placed. This means that we will create a forecast of the current signal of when we will get an opposite (closing) signal, and then make sure the profit potential between these two levels are enough for us to trade. I assume many readers are interested in how they can apply this to their own trading. Unfortunately this will differ from case to case, and the level of complexity will differ based on the indicator or trading technique. However, I will try to give you a more practical example that you hopefully find useful and can help you understand how to apply the technique in your own development:

Consider a strategy that is long when a moving average is pointing upwards, and short when it pointing downwards. We just received a long entry signal when the market breaks up from a range. This means that the moving average is more affected by the last fairly long breakout bar, compared to the previous bars with lower range. In order to get an exit (in this case short) signal, we will need the moving average to go down again with a distance larger than the previous small range bars (but not the same distance as our last long breakout bar). The trade will now have more room to move before it gets closed out at a profit or loss, compared with the size of the spread. Please note that this does not include any risk/reward measures, as the risk also increases as reward increases. Basically, we want to make sure that there is enough volatility in the market at our point of entry, and to make sure that we don’t exit the market too early due to low volatility so that the spread will cause most of the losses. If you’re interested in looking further into this with moving averages, I recommend you to consider the drop-off effect (the effect the last bar that is pushed outside the calculation period as the moving average shifts to a new bar, has on the value).

We’ll get back shortly with an update regarding the opportunity measures of the trading signal.

Johan Andreasson, System Investors

The synergy effects and how to lower sensitive variables

There can be a big advantage by combining different signals into a single trading system, such as different periods or alternative signals that aren’t too contradictory. While a trading system with only a few sets of conditions can have very sensitive variables, we can lower this risk to get a better overall performance by implementing more variables, or if different signals appear to have a synergy effect we can use a more aggressive position sizing during these signal to further increase our return. In the examples below we will use a moving average cross over system for illustrative purpose. Please note that these ideas might not give us an advantage with such a system, as this kind of system generally is unreliable without additional conditions. The system is chosen to simplify the explanation.

Decreasing the risk of sensitive variables

Consider a simple moving average cross over system, where we enter when a faster moving average crosses the slower moving average. This system can produce a good return in either trending markets or markets with big enough cycles, while it will empty your account during whipsaw periods. This kind of system can be very sensitive to the periods for the moving averages. If a 4/8 system (the faster moving average has period 4, while the longer moving average has period 8) produced a good return the last year, a system with 4/10 might have ended with a loss instead. However, the year after that it might be the opposite – the 4/10 produced a much higher return. Instead of choosing just one of these combinations, we can spread the investment into more instances of this system, but with different periods for the moving averages. This is actually basic diversification – to spread your risk, but we’re utilizing this in a single system instead of using multiple systems to lower our risk.

So, instead of opening a full position with the 4/8 system, we might choose to split this into 3 different combinations where we use different periods. One alternative might be to choose period combinations that represent different cycle lengths, to either benefit from moves within a trend/cycle or to follow the longer trend while the cycles seem to consolidate. In this case, we invest 33% of the position in a 2/4 system, 33% in the 4/8 system and the last in a 8/16 system. If you trading technique is good and has a positive expectancy, this is an easy method to get a smoother equity curve over a longer period.

Note: An alternative to this is simply to trade the same system multiple timeframes, to get a similar effect.

Take advantage of dependent probability

What is known as the synergy effect is actually the concept of dependent probability. Image you have two different signals – one that follow trends and one that act on swings in the market. Each of these methods has a 60% win ratio, which means that on average 60% of the trades are winners. However, if we use these two signals together such as we only trade when both systems gives the same signal, we suddenly have an 80% probability to close the trade with a profit! This is called the synergy effect; where a combination of two signals combined produce an outcome better than the two signals alone. In terms of probability, this is an example of dependent probability. Signal A has a 60% probability to be closed with a profit, but once we know that signal B has occurred, or probability is boosted to 80% instead.

This is actually what many traders do when they apply “filters” to their techniques, such as they only trade longs if the weekly trend is pointing upwards. However, this can be used more extensively by involving position sizing. If each signal alone already has a positive expectancy, we can use these for trading normal sizes. If a synergy signal occurs however, we have a much higher probability for a winning trade, thus we can add to our position to earn a higher return.

The reason why we bring up these concepts in this article, is because our development of the system has showed signs of both having somewhat sensitive variables and especially that it has a huge synergy effect that will be our biggest edge with this method. Even if we trade those signals that have an expectancy of breakeven, we can still benefit from these together with the other (higher probability signals) in the ways described above. I’ll try to explain:

Consider a signal A with 50% win ratio, where the size of the losers equals the winners. In addition, we have a signal B also with a 50% win ratio – but where the winners are twice as large as the losers. You might think that it’s completely unnecessary to trades signal A, since we don’t have a positive expectancy? WRONG! The reason is that is smoothes out the result, thus making us benefit from the compound effect even more if we base our position size on our equity. Consider the extremely simplified graphs below, where the first one shows signal A and the second one shows signal B. The trade sequence like this “never” happens, but the graphs have the ratios mentioned before so this is just to make the visualization easier.

The first graph simply shows the equity graph when fixed lot sizes are used (no cumulative effect). Even if this sequence is not very likely to occur, the point here is that if one strategy has a expectancy of 1 it won’t affect the result in the long run (if not spread, slippage and commission is taken into account):



The next graph shows to cumulative effect where we compare the equity graph of only signal B and then the equity graph of both signal A and B.



The second one is much smoother, as the movements are slightly cancelled out from the A signals with an expectancy of 1. To show that the concept is valid in more practical situations, please consider the following graph. This one is made up from a random streak of trades, where A had a expectancy of 1 and where B had an expectance of 1.5 (50% winners, where the winners were twice the losers). The simulation were run on 1000 iterations, and the average result of all is shown below. The dark line is the strategy which use the A signals together with B, while the gray is only the B signals:



As you can see a strategy with an expectancy of 1 will perhaps not give you any profits directly, but it will help you lower the risk in your strategy. If you get any synergy effect from you two (or more) strategies, you will definitely benefit from adding the signal A strategy.

Johan Andreasson, System Investors

Interesting Thread, hats off to you for your honest straightforward explanation of the process-
I believe you do make some valid points - ie using confluences of non correlated methods etc
I have been attempting to create an automated system myself however still in the testing stage, can I ask what software you are using .

All the best and good luck with it

excellent work my friend....Excellent work

let me know if you ever want to parle on forex correlation

NVP