Build Neural Network Indicator in MT4 using Neuroshell

[Krzysiaczek99]

Here I make mode detail investigation to verify the impact of neural net configuration in NS2. There are 16 available neural networks architecture, only Kohonen, PNN and GRRN are not implemented since two of them are used for classification purpose instead of prediction.

I tried to make prediction with all available nets and the minimum average error between 0.0059533 up to 0.0092735,except for GMDH net (applicable for one ouput) giving minimum average error 0.02418.

All results are MSE with 17 inputs and 3 outputs, the result will much better for 17 inputs and 1 outputs. Hence might be better to create 3 neural nets with separate training for 3 outputs prediction instead of one training for 3 nets at once. i will post the result afterward.

I tried to download Neurodimension and make some short prediction using the same data, this software giving minimum MSE at CV and Test are 0.04282 (higher than expected for a commercial software).


Arry

What was the input data ?? Is it data what we are using at the moment for MBP ??
(20kbars train/5 kbars test)?

Krzysztof

 

[arryex]

Kryzs,

Not yet, I am using my original NN trend indicator data. Please inform me direct link to get your data on which post no. The last one I put only for 52k but using NS predictor only. I 'll to investigate it.

Arry

 

[Krzysiaczek99]

Kryzs,

Not yet, I am using my original NN trend indicator data. Please inform me direct link to get your data on which post no. The last one I put only for 52k but using NS predictor only. I 'll to investigate it.

Arry

post 175. 20k training, 5k test

 

[Krzysiaczek99]

Maybe I don't understand what SS means. If we define deltaH as the change in High and deltapH as the change in the predicted high, then count the number of times the sign(deltaH) = sign(deltapH) and take the ratio of the count to the number of samples, we have SS the way I define it. That's what I did. Here is the modified spreadsheet that you posted. It shows my calcs. I think we agree on the definition of rmse. I just took the std(difference of desired and output).
View attachment 78100
I think that there may be small error in your function returning TRUE FALSE, maybe it returns False, True ?

I will do the same for the new report and edit this post later to give the result.
Edit:
It looks like you have:
Epoch SS RMSE (pips)
150K 0.58 23
300K 0.58 19
1M 0.6 18
See the sheets for the epochs above for explanation of how I got these.
View attachment 78104
fralo

Hi,

Explanation is attached. Open in Wordpad so it will not get reformatted.

Krzysztof

 

[fralo]

Hi,

Explanation is attached. Open in Wordpad so it will not get reformatted.

Krzysztof

Krzys,
I think we have a timing difference in calculation of the desired change and the predicted change. They are output simultaneously, so the changes must be computed simultaneously. See the attached explanation.
View attachment Krzys.zip
regards
fralo

 

[Krzysiaczek99]

Krzys,
I think we have a timing difference in calculation of the desired change and the predicted change. They are output simultaneously, so the changes must be computed simultaneously. See the attached explanation.
View attachment 78190
regards
fralo

If your interpretation is OK it means that it is just enough to shift network output
in my excel by 1 bar down to obtain your results. I did this and SS improved but still not to yours. Anyway i sent mail to Noel to confirm how we should interpret this. I hope I'm wrong and your results are correct

Krzysztof

 

[Krzysiaczek99]

I made another test of NN trying to predict direction only.

So i defined

DIR = 1 ==> price goes up, higher high
DIR = 0 ==> no change
DIR = -1 ==> price goes down, lower high

here are the results. I trained 15-45-45-15-1 net and 15-30-1 net, both 100k of epochs....Hard to believe how stabile result is regardless of net configuration.

I think finding the proper thresold value is a key here, NSDT does like this in it's
predictions

Krzysztof

 

[fralo]

I made another test of NN trying to predict direction only.

So i defined

DIR = 1 ==> price goes up, higher high
DIR = 0 ==> no change
DIR = -1 ==> price goes down, lower high

here are the results. I trained 15-45-45-15-1 net and 15-30-1 net, both 100k of epochs....Hard to believe how stabile result is regardless of net configuration.

I think finding the proper thresold value is a key here, NSDT does like this in it's
predictions

Krzysztof

Have you got Noel's response yet?

If we use the average of the predictions as the threshold, then
Using this data and a linear perceptron we get SS = 0.64112337 (topology = 15-1)
Using the 15-30-10 net, we will get SS = 0.655967904
Not much difference. The Perceptron gets most of the information in the data set, more weights just gives slight improvement.

Of course this is not fair, because in real time we will not have the average of the predictions (non causal), but using a fairly long SMA (that is causal) does not make a big difference in SS.

Can you somehow feed this data to NSDT to get a threshold? What threshold does NSDT find? Is it causal?
Can NSDT make a set of rules for this data? Sorry for all the questions, but I have not yet mastered NSDT..have concentrated on MBP, since it generates useful results and nets that I understand.
View attachment Output.zip

edit: Somewhere in one of your posts you gave a reference to a paper that said that SS>=.6 was useful. I cannot find it. Can you direct me to it?
fralo

 

[Krzysiaczek99]

Have you got Noel's response yet?

If we use the average of the predictions as the threshold, then
Using this data and a linear perceptron we get SS = 0.64112337 (topology = 15-1)
Using the 15-30-10 net, we will get SS = 0.655967904
Not much difference. The Perceptron gets most of the information in the data set, more weights just gives slight improvement.

Of course this is not fair, because in real time we will not have the average of the predictions (non causal), but using a fairly long SMA (that is causal) does not make a big difference in SS.

Can you somehow feed this data to NSDT to get a threshold? What threshold does NSDT find? Is it causal?
Can NSDT make a set of rules for this data? Sorry for all the questions, but I have not yet mastered NSDT..have concentrated on MBP, since it generates useful results and nets that I understand.
View attachment 78232

edit: Somewhere in one of your posts you gave a reference to a paper that said that SS>=.6 was useful. I cannot find it. Can you direct me to it?
fralo

Yes I can feed this data to NSDT and yes it can search for most profitable rules. It was my plan actually to make indicators now either for MC or NSDT. The advantage of MC is that it has multi threaded optimizer so much faster but it needs more programming
so i think better will be to use NSDT in this case.

if you make NN predictions in NSDT it finds thresolds for buying and selling signals.

I just realized what actually Ward its doing with those NN predictions. I think they dont train it very much but for the speed they optimize the structure using GA. Even when you start training or what ever it is callled, it says 'optimizing'. You can see in Arrys reports from predictor that they give 'best net statistic' giving optimal number of neurons. So i think those nets are huge, learn fast with huge dataminig bias error so bad performance in OOS.

I think the best net what he shown for 7k file had 135 neurons and for 52k file 61.

i pointed to this 0.6 becouse i think than at least 60% of trades must be done on the correct side.

No, no reply from Noel yet. Than I will try to import data to NS and make some dlls.

Krzysztof

 

[fralo]

Yes I can feed this data to NSDT and yes it can search for most profitable rules. It was my plan actually to make indicators now either for MC or NSDT. The advantage of MC is that it has multi threaded optimizer so much faster but it needs more programming
so i think better will be to use NSDT in this case.

if you make NN predictions in NSDT it finds thresolds for buying and selling signals.

I just realized what actually Ward its doing with those NN predictions. I think they dont train it very much but for the speed they optimize the structure using GA. Even when you start training or what ever it is callled, it says 'optimizing'. You can see in Arrys reports from predictor that they give 'best net statistic' giving optimal number of neurons. So i think those nets are huge, learn fast with huge dataminig bias error so bad performance in OOS.

I think the best net what he shown for 7k file had 135 neurons and for 52k file 61.

i pointed to this 0.6 becouse i think than at least 60% of trades must be done on the correct side.

No, no reply from Noel yet. Than I will try to import data to NS and make some dlls.

Krzysztof

Yes, I read in the tutorial that NSDT adds nodes one at a time as it optimizes. It is the opposite of the pruning technique.

I think that when you feed data to NSDT, you should have one net to predict high direction and one to predict low direction. Then if both are up the rule will be long, if both down the rule will be short. Else hold. Even if the predictions are correct, there is possibility of loss, so 0.6 may not be good enough. More sophisticated rules may be required.

Meanwhile I will try to improve SS using a quasi-stationary data base. I think I can do better than .6
fralo

 

[arryex]

Hi Guys,

Sorry not participating for a while..I am on travelling with my family..

Here is my investigation result "how to create Neural Network using Genetic Optimization". There are some advantages using GA:
- The training always made out of sample for all given data, hence no worry for the curvefit.
- The output optimization can be defined as you wish either r-squared, average absolute error, MSE, RMSE etc.
- I plan to use GA as well to optimize the trade entry/profit calculation but need additonal learn.
- Network architecture can be created as you wish, more complex network more calcualation size.

Here I give 1st page a free version, the pdf version is password protected. As usual the password only given to the person supporting this thread (please PM me).

Cheers,
Arry

 

[arryex]

I failed to post the excel file even on the zip format, the excel is stil more than 6 MB.

Anyone who nee the Excel file for NN with 3 layer, 5 hidden Neurons sample result Please PM me, I wil sent to your email (file is more than allowed size attachment).


Arry

 

[fralo]

Hi Guys,


- The training always made out of sample for all given data, hence no worry for the curvefit.
- The output optimization can be defined as you wish either r-squared, average absolute error, MSE, RMSE etc.
- I plan to use GA as well to optimize the trade entry/profit calculation but need additonal learn.


Cheers,
Arry

Hi Arry.
I fail to see how GA optimized over the whole sample avoids curve fitting. Seems like you need to divide your data into a set for optimization and another for testing.Otherwise you just test over what you have optimized.
fralo

 

[Krzysiaczek99]

Hi Guys,

Sorry not participating for a while..I am on travelling with my family..

Here is my investigation result "how to create Neural Network using Genetic Optimization". There are some advantages using GA:
- The training always made out of sample for all given data, hence no worry for the curvefit.
- The output optimization can be defined as you wish either r-squared, average absolute error, MSE, RMSE etc.
- I plan to use GA as well to optimize the trade entry/profit calculation but need additonal learn.
- Network architecture can be created as you wish, more complex network more calcualation size.

Here I give 1st page a free version, the pdf version is password protected. As usual the password only given to the person supporting this thread (please PM me).

Cheers,
Arry

I have to agree with fralo comment. It's not an OOS test. Even pointing the best net for OOS its not OOS test because you are choosing the best one from a group so its calibration/paper trading. We want here to evaluate predictive power so test mus be done on really unseen data so e.g. when you choose the best net performing OOS (!!!Calibration) than you have to test its performance on unseen data to be true OOS test.

Krzysztof

 

[arryex]

It is easy to adjust the range for training/finding the weight neruons, no problem . Sorry if my statement is misslead.

For your information, using GA in excel/other software do not use any calibration, as usual we can split the range as training, test and OOS. All weight parameters are updated (almost same as other excel add in), the correct word shall be 'evolving' instead of optimization.

There is one very expensive product ($ 60KUSD + 20K USD only for 2 years!!!) TSL, see http://www.tradingsystemlab.com.

The algorithme might be similar with TSL, i.e. using GA evolving methods.
"Genetic algorithms (GAs) seek to solve optimization problems using the methods of evolution, specifically survival of the fittest. If the model is a formula, then we will usually be seeking the maximum or minimum value of the formula."

To avoid curve fitting (mentioned on TSL description):
- First, the evolved Trading Systems have their size pruned down to the lowest possible size through what is called Parsimony Pressure, drawing from the concept of minimal description length. Thus the resultant Trading System is as simple as possible and it is generally believed that the simpler the Trading System is, the better it will perform into the future. (I am little confuse?)
- Secondly, randomness is introduced into the evolutionary process, which reduces the possibility of finding solutions that are locally, but not globally optimum. Randomness is introduced over not just the combinations of the genetic material used in the evolved Trading Systems, but in Parsimony Pressure, Mutation, Crossover and other higher-level GP parameters. Out of Sample testing is performed while training is in progress with statistical information presented on both the In Sample and Out of Sample Trading System testing.

Arry

 

[Krzysiaczek99]

It is easy to adjust the range for training/finding the weight neruons, no problem . Sorry if my statement is misslead.

For your information, using GA in excel/other software do not use any calibration, as usual we can split the range as training, test and OOS. All weight parameters are updated (almost same as other excel add in), the correct word shall be 'evolving' instead of optimization.

There is one very expensive product ($ 60KUSD + 20K USD only for 2 years!!!) TSL, see http://www.tradingsystemlab.com.

The algorithme might be similar with TSL, i.e. using GA evolving methods.
"Genetic algorithms (GAs) seek to solve optimization problems using the methods of evolution, specifically survival of the fittest. If the model is a formula, then we will usually be seeking the maximum or minimum value of the formula."

To avoid curve fitting (mentioned on TSL description):
- First, the evolved Trading Systems have their size pruned down to the lowest possible size through what is called Parsimony Pressure, drawing from the concept of minimal description length. Thus the resultant Trading System is as simple as possible and it is generally believed that the simpler the Trading System is, the better it will perform into the future. (I am little confuse?)
- Secondly, randomness is introduced into the evolutionary process, which reduces the possibility of finding solutions that are locally, but not globally optimum. Randomness is introduced over not just the combinations of the genetic material used in the evolved Trading Systems, but in Parsimony Pressure, Mutation, Crossover and other higher-level GP parameters. Out of Sample testing is performed while training is in progress with statistical information presented on both the In Sample and Out of Sample Trading System testing.

Arry

Grail does the same and much cheaper - CASB - Computer assisted strategy builder.
Uses GA to find strategies. Randomness CASB also uses its called stress. Additionaly you can import your own indicators to CASB, im not sure if you can using this tool.

However when I asked for test report of CASB i.e. how the predictive power of this
was tested author told me its confidential... because he worked for a fund before and is not allowed to give any info.....

All those optimization based methods introducing huge data mining bias error,
if this tool is so expensive that they should provide test report of predictive power
- you can ask just wonder what answer will be...

Krzysztof

 

[arryex]

Can we use CASB to find NN parameter using its GA? or only searching the parameter for deterministi indicator parameter (RSI, CCI, etc) to build profitable trade.

Using my excel, I may use it to mazimize the profit and use indicator parameter as chromosomes. I wil have a look.

About TSL? I am very sure that I will not buy it ha..ha.. except my profit reach there...my concern is how they can make it...

Arry

 

[kijkij]

Arryex!
Interesting thread found.

Thanks MadCow for referring me this thread.

Just understanding neural network. Trying to figuring out how to use SNNS and MBP. The problem is MBP have many bugs and SNNS is looks complicated to understand. So any one here can explain the use of SNNS. I have read the SNNS user manual but not sure what they are saying.

 

[arryex]

Kijkij, Wellcome to the thread. I am sure we will get benefits from what we are discussed.

I would like to remind that even a profitable strategy is our main objective, I believe that understanding the problem, identify what data we have, what we intend to predict, what software intend to be used, which method to train the network, how to deploy trained net are parts of the job prior getting a profitable trade..

Madcow and Kryzs have started with MBP, hope you can start showing your finding about SNNS and any body else can assist you.

Arry

 

[kijkij]

Kijkij, Wellcome to the thread. I am sure we will get benefits from what we are discussed.

I would like to remind that even a profitable strategy is our main objective, I believe that understanding the problem, identify what data we have, what we intend to predict, what software intend to be used, which method to train the network, how to deploy trained net are parts of the job prior getting a profitable trade..

Madcow and Kryzs have started with MBP, hope you can start showing your finding about SNNS and any body else can assist you.

Arry

Thanks Arry,

1. Upto this point what knowledge I have, just want to ask what if we simply train NN on Heiken Ashi? Input the High, Low and Close from the HA of the previous 21 or 34 bars and predict next 1 bar as ouput! is it not batter than normal prices?

2. Is there a difference if I train a network on 20,000 patterns than just 1000 patterns?

Kashif