# Ensemble neural network

I had run artificial neural network on Matlab. Although i used the same design structure of ANN and the same data set, the result always different. Some suggested using ensemble neural network. From my reading ensemble is combine ANN with different design structure. Do this applicable in my problem. Other than that, why is ANN produce different result every time i run it?

Why ANN produces different results: this is probably the training procedure involves "randomness", for example,

1. Your training may use random parameter initialisation.
2. Your training may use random dropout as regularisation.
3. Your training may use SGD and shuffle the data order every epoch.
4. ...

And ANN is a not a convex function, so any of these "randomness" may lead to different local optimum.

And ensemble method is a very general method to reduce the variance of predictor then improve the performance. It is not necessarily specific for ANN. And there are many ways to train an ensemble of ANN (like using different datasets, using different initial parameters, using different structure/dropout). This is really an "art" which means you need to try a lot and pick the best way for your specific dataset.

• If i only have one data set, Can i only take the average of the output? Jan 19 '17 at 0:33
• @bbadyalina you can try different parameter initialisation. You can also obtain multiple datasets by bootstrapping the one dataset you have. Just to remember that, if you are using averaging ensemble method, try to "decorrelate" the different models as much as possible (i.e. make them as noncorrelated/independent as possible). Jan 19 '17 at 0:50

Your second question is more simple so I'll answer that first.

When you train an ANN you start with random weights in the network. The network is trained to try to minimize the cost function of training. We usually cannot find the exact minimum of the cost function during training so each time you train an ANN it starts with different weights and ends training with different weights.

Training a single ANN is quite costly so it could be difficult to train 10 ANNs and make an ensemble of them all. Instead we use a clever trick called dropout when training one ANN, dropout is a type of regularization which acts like training an ensemble of networks inside one network.

Here is a description of dropout, scroll down to read about dropout.

• So if train the network ten times using the same structure, and take the average of the output. Is that reliable result? Jan 19 '17 at 0:35
• @bbadyalina Yes if you have the computing power to train a network 10 times then taking the average output is more accurate. You should use the standard deviation of the output to create a confidence interval and use that to understand how reliable the average is.
– Hugh
Jan 19 '17 at 14:09
• ok hugh.. is there any reference that i can use for this method.. or it is the same as ensemble ANN.. from what i read ensemble ANN is combine different sturucture of ANN. but i did not find any of regrading single structure. Jan 19 '17 at 14:12
• @bbadyalina The ensemble will work if you use the same structure for all ANN's but it helps if you use different structures. I don't know any references for choosing the different structures
– Hugh
Jan 19 '17 at 15:01
• If for same structure, do have any reference? Jan 19 '17 at 15:02