I'm looking for some advice on a general approach to optimise the training of a neural network. My primary concern is to avoid over-fitting to the training data and maintain as much generality as possible.

I'm using the Resilient Backproppgation method and as such need to optimise:

  • number of epochs
  • learning rate
  • network topology (number of neurons in hidden layer)

I had in my mind that I could perform the following process:

Load LabelledData.csv

Foreach epoch


// Randomly split labelled data into 90% training and 10% test

// Backprop the training data

// Calculate error against test data


plot(epoch, error)

// Iterate number of neurons, epochs and learning rate then repeat

The drawback with this method is that over time, the network is trained with all the data since a portion is never removed absolutely before training. If I was to remove the test set prior to beginning training, how could I perform a, say, k-fold cross validation, to ensure that the test set was representative of the training data?

Thank you for your help!


1 Answer 1


Performing k-fold cross validation can be computationally expensive for complex neural networks, but if you can do it you should.

If you choose do k-fold cross validation you have to train k distinct neural networks keep track of the validation error across different networks and epochs. I suppose after you could plot how the mean and standard deviation of the error (across k-folds) evolves with the number of epochs, and determine the optimal number of epochs with which you should train with all the data to get a final network.

If performing k-fold cross validation is too expensive you could use a technique called early stopping where you stop training when you performance on the validation set (just one) starts to deteriorate.

I have never seen people changing the train and validation set between epochs in the same training instance like you suggested, I am not sure it is advisable to do so.

  • $\begingroup$ Thank you, Miguel. So my method will look something like this: split data into k folds. For each fold, backprop the training data, stopping when the validation error increases. Grid search the learning rate and number of neurons for each fold. Now average the optimum learning rate and number of neurons found for each fold. I could skip this step by taking an educated guess at both parameters. Then do k fold CV again as you describe to determine the optimum number of epochs to train the full set over? $\endgroup$
    – tommyzer00
    Commented Jul 23, 2017 at 8:55
  • $\begingroup$ That sounds like a solid plan, even though it will take a lot of training... Don't forget to train one last time with all the data and the parameters you determined. I would say that you don't need to be changing the number of neurons in your network, just use regularization (l1 or l2) and adjust the regularization parameter as you do the learning rate, it is easier and faster. $\endgroup$
    – Miguel
    Commented Jul 23, 2017 at 9:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.