Optimising neural network to prevent overfitting

Question

I'm looking for some advice on a general approach to optimise the training of a neural network. My primary concern is to avoid over-fitting to the training data and maintain as much generality as possible.

I'm using the Resilient Backproppgation method and as such need to optimise:

• number of epochs
• learning rate
• network topology (number of neurons in hidden layer)

I had in my mind that I could perform the following process:

Load LabelledData.csv

Foreach epoch

{

// Randomly split labelled data into 90% training and 10% test

// Backprop the training data

// Calculate error against test data

}

plot(epoch, error)

// Iterate number of neurons, epochs and learning rate then repeat

The drawback with this method is that over time, the network is trained with all the data since a portion is never removed absolutely before training. If I was to remove the test set prior to beginning training, how could I perform a, say, k-fold cross validation, to ensure that the test set was representative of the training data?

Thank you for your help!

Answer 1

Performing k-fold cross validation can be computationally expensive for complex neural networks, but if you can do it you should.

If you choose do k-fold cross validation you have to train k distinct neural networks keep track of the validation error across different networks and epochs. I suppose after you could plot how the mean and standard deviation of the error (across k-folds) evolves with the number of epochs, and determine the optimal number of epochs with which you should train with all the data to get a final network.

If performing k-fold cross validation is too expensive you could use a technique called early stopping where you stop training when you performance on the validation set (just one) starts to deteriorate.

I have never seen people changing the train and validation set between epochs in the same training instance like you suggested, I am not sure it is advisable to do so.