Build Neural Network Indicator in MT4 using Neuroshell

[fralo]

I hope it will work. I used COUNTIF which is new but it said that it is converted.

Thanks Krzys, works OK.
I think that the stdev calcs average over future instead of past. (Though it doesn't make a difference if interpreted correctly, so either way.) Also, if you use maybe 500 samples for stdev you will get smoother curve that more clearly shows non stationarity.

Do you conclude that the net architecture is not important. Or do you come to some other conclusion? Maybe there are too many sign errors to be useful?

MadCow

 

[fralo]

Seems to be suspicious result. The sign match ratio is .43. If you reversed sign of output, you would have SS% = .57. This might be useful! What gives here? Is this an error because the sample size is too small, or has the net actually extracted some information, assbackwards though it may be?

MadCow

 

[arryex]

The range of error given is -0.01156 up to 0.012219 for 15-30-10-1 Net and -0.01073 up to 0.015321 for 15-30-1 net.

Short question:
- adding new layer can improve prediction (less error), is it possible to add more layer and see the result?
- Is it correct to accept 0.012219 error which is 122 pips?

Arry

 

[Krzysiaczek99]

Hi,

Yes I have some conclusions. First RMS errors in test period

15-30-1 21pips
15-30-10-1 16.5pips
15-30-30-1 22pips

first 2 errors were achieved on 150k epochs, last one on 200k epochs. In all cases error was still falling so is possible to obtain even lower one. Adding neurons dont improve significiantly then but increases the time of training. Last net took more than 12h to make 150K epochs.

other observations and conclusions

Propotion between IS/OOS error is like 1/1.25

than i will try to run 15-30-10-1 even more to see how much down we can get but error falls very slowly now

Its hard to believe than ward nets are so efficient in learning comparing to MBP, i think there must be some price for this in Ward nets, they just made NS2 and Predictor to be able to use with PCs made 10 years ago.

Regarding SS. For my EURUSD NSDT prediction it was shown around 50%, hope its calculated the same way like i did. I think usefull value would be like > 60% and with target of 80%. i dont know if it is possible but if we train till very low RMS SS should start to improve drastically. Also SS increases when sample size decrease but i dont know what will happen to RMS in this case. So in next stage i will cut training size by 2 to 10k bars

I think we are most intrested in absolute errors when SS is TRUE and FALSE but separatelly. If its TRUE we are on the correct side of trade but when FALSE we are on the opposite. They can be quite different. Analysis of SS in time would be also interesting.

Regarding Arry question about big errors. Of course is not correct but analysis of errors can tell where to put stoploss

Krzysztof

 

[fralo]

Its hard to believe than ward nets are so efficient in learning comparing to MBP, i think there must be some price for this in Ward nets, they just made NS2 and Predictor to be able to use with PCs made 10 years ago.
Krzysztof

Ward nets are not magic. I think NS2 stops training when RMSE on validation set starts to increase. This usually avoids over training (curve fitting) and improves generalization. You can do the same with MBP, if you split data sets into 3 parts, (Train, Test, OOS) and use train and Test in MBP, then train until Test RMSE begins to increase (too bad you must do this manually, it should be added as a feature) then you have training like NS2 and NSDT, and it will be very quick since epochs will be few. Then you save code and use it to test performance on OOS set (Doing this in MBT would be another desirable feature).

Training for many epochs very often decreases performance on OOS, because it is overtraining on Train set and can curve-fit if many weights are available.Then net will not generalize as well. This is a major problem with NN's.
Two solutions: Use a Validation set as above, and/or use as few weights and many data samples as possible.

Regarding SS. For my EURUSD NSDT prediction it was shown around 50%, hope its calculated the same way like i did. I think usefull value would be like > 60% and with target of 80%. i dont know if it is possible but if we train till very low RMS SS should start to improve drastically. Also SS increases when sample size decrease but i dont know what will happen to RMS in this case. So in next stage i will cut training size by 2 to 10k bars

Krzysztof

In your analysis of the MBT trained net (15-30-xx-report.xls) you have an SS of 0.435748599. That suggests that if the signs were reversed for each output, the SS would be ~.564, which is approaching 60%. However, the sign inversion makes me think that something funny is going on here. Maybe the result is just luck. Here are curves from the data in 15-30-xx-report.xls for the SS averaged over 500 samples and the rmse averaged over 500 previous samples.

You can see that the SS is changing only at the end. The RMSE changes also at the end, going from about 4-5 pips to ~ 7 pips at the end. This indicates non-stationarity, and suggests that we need to look at other data. When I first posted this data, I was trying to replicate Arry's post (4, I think), and did not mean that we should study it so closely. I thought then and continue to think that we should look at a more stationary set of data. The data set we are using has values in the OOS set that do not occur in the Train set. If you teach a child the alphabet from A to M, you cannot expect him to recognize the letters X, Y, Z. Nets are a lot like children. You must train them with data that covers all possibilities. This requires stationary, or quasi stationary data such as the first differences of prices, or else it requires painstaking effort to be sure that the training set is truly representative of all data that might be encountered.

I will post a data set that is quasi stationary when we are done with this if you all want one.

MadCow

 

[fralo]

The range of error given is -0.01156 up to 0.012219 for 15-30-10-1 Net and -0.01073 up to 0.015321 for 15-30-1 net.

Short question:
- adding new layer can improve prediction (less error), is it possible to add more layer and see the result?
- Is it correct to accept 0.012219 error which is 122 pips?

Arry

Arry,
Adding a layer may improve prediction, but sometimes at the cost of loss of generalization. You must carefully evaluate net operation on OOS data, and decide whether the added weights have really helped.

I think that the error you quoted is the RMS error reported by MBP. MBP does not report errors in pips. You must run the net using the test data by saving the code and running it against the test file. I think that's what Krzys did in his report. That shows an error that is ~ 7 pips (see last post). Although 7 pips is pretty good, I do not trust the net because the SS is .44, and the data is non-stationary.
fralo/MadCow

 

[Krzysiaczek99]

Arry,
Adding a layer may improve prediction, but sometimes at the cost of loss of generalization. You must carefully evaluate net operation on OOS data, and decide whether the added weights have really helped.

I think that the error you quoted is the RMS error reported by MBP. MBP does not report errors in pips. You must run the net using the test data by saving the code and running it against the test file. I think that's what Krzys did in his report. That shows an error that is ~ 7 pips (see last post). Although 7 pips is pretty good, I do not trust the net because the SS is .44, and the data is non-stationary.
fralo/MadCow

Yes, this what im doing.you can save the data from MBP just right clicking on the chart.
Than you can copy to excel.

Now I trained till 250k epochs for 15-30-10-1 and error still falls

training 0.001399
testing 0.001531

so it improved for OOS for 1.2 pip. Factor of errors testing/training 1.09 so still very good and constant. Than i will continue with training

By ward nets that they are tricky i meant that they need very little training, just have a look to the Arrys report. Factor of errors for NSDT prediction with Fisherized data
for 6k bars file was 1.18 so a bit higher and input data was stationary (Momentum)

Krzysztof

 

[fralo]

Yes, this what im doing.you can save the data from MBP just right clicking on the chart.
Than you can copy to excel.

Now I trained till 250k epochs for 15-30-10-1 and error still falls

training 0.001399
testing 0.001531

so it improved for OOS for 1.2 pip. Factor of errors testing/training 1.09 so still very good and constant. Than i will continue with training

By ward nets that they are tricky i meant that they need very little training, just have a look to the Arrys report. Factor of errors for NSDT prediction with Fisherized data
for 6k bars file was 1.18 so a bit higher and input data was stationary (Momentum)

Krzysztof

Right click is a lot easier than what I've been doing, but where does MBP put the file, and what does it name it?

 

[Krzysiaczek99]

Right click is a lot easier than what I've been doing, but where does MBP put the file, and what does it name it?

it puts to the clipboard. There are tutorials video on MBP site so i know.....

 

[Krzysiaczek99]

here is a data after 300k epochs for 15-30-10-1. Error still falls. Perhaps till end of the week I will reach 5pips error . So because of falling error SS should rise, i haven't checked yet. If it doesn't than something is wrong.

Krzysztof

 

[fralo]

here is a data after 300k epochs for 15-30-10-1. Error still falls. Perhaps till end of the week I will reach 5pips error . So because of falling error SS should rise, i haven't checked yet. If it doesn't than something is wrong.

Krzysztof

SS=0.577631052 RMSE = 0.001288988
fralo

 

[Krzysiaczek99]

SS=0.577631052 RMSE = 0.001288988
fralo

Intresting, MBP shows RMSE for this data a bit different. See screen. I will run it overnight and see where it will end.

Krzysztof

 

[arryex]

Congratulation..a good progress...

Arry

 

[Krzysiaczek99]

I'm not so sure about this progress...but its something.

Here are all MBP nets:

15-30-1
15-30-30-1
15-10-10-1

15-30-10-1 was tested gradually, stopped, test resumed etc till 1 million epochs. So you can open them and grab data by right clicking for analysis.

Arry, did you checked impact of Fisherizing of price on your nets maybe ?? Does it help ?? Did you try with another bigger data sets ??

Fralo. Can you post your data with dates ?? I think it was posted once but it is not correct file i think, should have total 25k bars, it has like 5.5k

I will try to train 15-30-10 but from 10k not 20k training file. Lets see error...

Krzysztof

 

[fralo]

I'm not so sure about this progress...but its something.

Here are all MBP nets:

15-30-1
15-30-30-1
15-10-10-1

15-30-10-1 was tested gradually, stopped, test resumed etc till 1 million epochs. So you can open them and grab data by right clicking for analysis.

Arry, did you checked impact of Fisherizing of price on your nets maybe ?? Does it help ?? Did you try with another bigger data sets ??

Fralo. Can you post your data with dates ?? I think it was posted once but it is not correct file i think, should have total 25k bars, it has like 5.5k

I will try to train 15-30-10 but from 10k not 20k training file. Lets see error...

Krzysztof

You are quite right. I don't know what happened. :whistling Here is the correct file. Note there is a gap in dates between train and test data to make sure that tests were on truly independent data. There are some test values that are not in the training set.

My experience with this data and MBP is that the value of SS and RMSE certainly get better with more training data, weights, and epochs, but the difference is quite small. I think I will try a linear perceptron to see if the nonlinearity is really very helpful. I find that I can speed training a lot if I start training in the on-line mode, and after the RMSE settles, then stop and switch to the batch mode. I think that CUDA requires batch mode, so that won't help you much except for quick tests of topologies.

edit: The perceptron will not do well with RMSE, but it gives an SS of 0.545218087
How much is luck? It would be interesting to plot SS vs some measure of net complexity.. maybe number of weights. The perceptron requires 15 weights. I don't think we will get to an SS of .6 with this data set.
fralo
View attachment DatedEURUSDH1.zip

 

[Krzysiaczek99]

Strange that you are getting so good value of SS. Is it with my data and with my excel sheet ?? See my full report. I added data from other nets i.e. 15-30-1 and 15-30-30-1
and test results after different number of epochs for 15-30-10.

Average Stdev of error very stabile, SS only grows a bit for 15-30-30 but this one has bigger RMSE in this case. I think in order to predict properly SS we should use SS as predict objective not H so like a fitness function. Than maybe two or more nets will necessary, one for prediction of SS and another for prediction of high.

I train now on 10k bars - it works much faster but error much bigger. It reached 420k epochs and RMS from MBP 23 pips.

Krzysztof

 

[arryex]

Here I make mode detail investigation to verify the impact of neural net configuration in NS2. There are 16 available neural networks architecture, only Kohonen, PNN and GRRN are not implemented since two of them are used for classification purpose instead of prediction.

I tried to make prediction with all available nets and the minimum average error between 0.0059533 up to 0.0092735,except for GMDH net (applicable for one ouput) giving minimum average error 0.02418.

All results are MSE with 17 inputs and 3 outputs, the result will much better for 17 inputs and 1 outputs. Hence might be better to create 3 neural nets with separate training for 3 outputs prediction instead of one training for 3 nets at once. i will post the result afterward.

I tried to download Neurodimension and make some short prediction using the same data, this software giving minimum MSE at CV and Test are 0.04282 (higher than expected for a commercial software).


Arry

 

[arryex]

Hi Krizys,

Here is the ful report for 52k data as per your request..I made it with NS predictor.

Arry

 

[fralo]

Strange that you are getting so good value of SS. Is it with my data and with my excel sheet ?? See my full report. I added data from other nets i.e. 15-30-1 and 15-30-30-1
and test results after different number of epochs for 15-30-10.

Average Stdev of error very stabile, SS only grows a bit for 15-30-30 but this one has bigger RMSE in this case. I think in order to predict properly SS we should use SS as predict objective not H so like a fitness function. Than maybe two or more nets will necessary, one for prediction of SS and another for prediction of high.

I train now on 10k bars - it works much faster but error much bigger. It reached 420k epochs and RMS from MBP 23 pips.

Krzysztof

Maybe I don't understand what SS means. If we define deltaH as the change in High and deltapH as the change in the predicted high, then count the number of times the sign(deltaH) = sign(deltapH) and take the ratio of the count to the number of samples, we have SS the way I define it. That's what I did. Here is the modified spreadsheet that you posted. It shows my calcs. I think we agree on the definition of rmse. I just took the std(difference of desired and output).
View attachment return15-30-10-1_test_300k_epochs.zip
I think that there may be small error in your function returning TRUE FALSE, maybe it returns False, True ?

I will do the same for the new report and edit this post later to give the result.
Edit:
It looks like you have:
Epoch SS RMSE (pips)
150K 0.58 23
300K 0.58 19
1M 0.6 18
See the sheets for the epochs above for explanation of how I got these.
View attachment Return15-30-xx-report.zip
fralo

 

[arryex]

This is my personal review concerning NN design...any comment/advice are appreciated..

Arry