In an attempt to find the optimal number of hidden nodes and number of training epochs (to obtain optimal performance while not over-fitting) in a single-hidden layer neural network, I generated the following figure.

Levenberg-Marquardt MSE performance

The application is function approximation of temperature data and the error measure displayed in the figure is therefore mean square error (MSE) in degree Celsius squared. I used the Levenberg-Marquardt training algorithm and split the dataset into training(70%)-validation(15%)-test(15%) subsets. The neural network has 5 inputs and 1 output.

In Fig. a) the MSE falls to approximately 0.129 (training) and 0.144 (validation) at n=5 and remains there, meaning the network weights and biases have converged. However, in other applications, the validation error typically starts to increase at a given training epoch n (see for example Fig.1 in this paper). In such cases training should stop before this value of n. Why doesn't Fig. a) show this trend? Is there a neural network theorem or guideline stating what behaviour can be expected for different types of applications?

Similarly, in Fig. b) it seems that all errors continue to decrease. How should the optimal number of hidden nodes be selected given such a graph? Should I evaluate the number of hidden neurons at much higher values? (beyond $N_h$=98 I run into memory problems...).


The reason that the validation error does not increase at some point in Fig. a) is that overfitting does not occur - as the training data size is much larger than the trainable parameters in the network. On p. 473 of Neural Network Design it is stated that if "the number of data points is significantly larger than the number of network parameters, then there will not be a problem of overfitting." There is therefore no "guarantee" that the validation error will start to increase at some point.

Also, by increasing the number of hidden nodes in Fig. b) beyond $N_h$= 98, the training error continues to decrease, while the validation and testing error levels off (does not increase or decrease further).


Not the answer you're looking for? Browse other questions tagged or ask your own question.