# How do I improve my neural network stability?

I'm using the neuralnet in R to build a NN with 14 inputs and one output. I build/train the network several times using the same input training data and the same network architecture/settings.

After each network is produced I use it on a stand alone set of test data to calculate some predicted values. I'm finding there is a large variance in each iteration of the predicted data, despite all the inputs (both the training data and test data) remaining the same each time I build the network.

I understand that there will be differences in the weightings produced within the NN each time and that no two neural networks will be identical, but what can I try to produce networks that are more consistent across each train, given the identical data?

• Can you give us a little bit more detail on the learning algorithm and architecture that you (or the R package) used? How many layers does the NN have? – Lucas Feb 21 '12 at 16:42
• Hi Lucas, I'm using the R package neuralnet link which has a good explanatory article here link. I'm using one hidden layer of 8 neurons. The learning algorithm is resilient backpropagation with weight backtracking. – tfb Feb 22 '12 at 8:32

## 2 Answers

In general you would get more stability by increasing the number of hidden nodes and using an appropriate weight decay (aka ridge penalty).

Specifically, I would recommend using the caret package to get a better understanding of your accuracy (and even the uncertainty in your accuracy.) Also in caret is the avNNet that makes an ensemble learner out of multiple neural networks to reduce the effect of the initial seeds. I personally haven't seen huge improvement using avNNet but it could address your original question.

I'd also make sure that your inputs are all properly conditioned. Have you orthogonalized and then re-scaled them? Caret can also do this pre-processing for you via it's pcaNNet function.

Lastly you can consider tossing in some skip layer connections. You need to make sure there are no outliers/leverage points in your data to skew those connections though.

• Interestingly I switched the training of the net to the 'nnet' function (available in the package of the same name) and the results on the test set of data have become much more stable - maybe something to do with the different way the weights are initialized between the two packages? – tfb Feb 22 '12 at 12:36
• In the nnet the initial weights are all initialized to a uniform random number between -0.7 and 0.7 if I recall correctly. And you can control the magnitude in a parameter. I've honestly had solid luck with the nnet package and have never tried any of the other options. Best of luck! – Shea Parkes Feb 22 '12 at 13:26

I haven't worked with R, so I can only give more general tips.

Did you check whether the algorithm converged? One possible explanation might be that the different parameter sets are all somewhere half way to the same optimum.

If the algorithm always converges but to a different local optimum, then there are many heuristics you could try to avoid those. One simple strategy when using stochastic gradient descent (SGD) would be to use smaller batches and larger momentum. The smaller batch sizes effectively introduce some noise into the training which can help escape some local optima. A much more sophisticated strategy would be to initialize the weights using autoencoders.

• Just as an fyi, as long as he's using the nnet from base R, it uses the BFGS optimization method from R's optim. It actually calculates the gradients to get a picture of the surface. There is no batch processing and no fixed momentum parameter in it's implementation. Having said all that, it can easily fail to converge; especially with garbage in. – Shea Parkes Feb 21 '12 at 19:41
• @SheaParkes thanks for the comments and answers Shea. I'm actually using the neuralnet package - see comment above. It uses a resilient backpropagation algorithm with weight backtracking – tfb Feb 22 '12 at 8:40
• Then my apologies Lucas, I missed that tidbit. I'm glad you got it worked out tfb. – Shea Parkes Feb 22 '12 at 13:25