In an attempt to find the optimal number of hidden nodes and number of training epochs (to obtain optimal performance while not over-fitting) in a single-hidden layer neural network, I generated the following figure.
The application is function approximation of temperature data and the error measure displayed in the figure is therefore mean square error (MSE) in degree Celsius squared. I used the Levenberg-Marquardt training algorithm and split the dataset into training(70%)-validation(15%)-test(15%) subsets. The neural network has 5 inputs and 1 output.
In Fig. a) the MSE falls to approximately 0.129 (training) and 0.144 (validation) at n=5 and remains there, meaning the network weights and biases have converged. However, in other applications, the validation error typically starts to increase at a given training epoch n (see for example Fig.1 in this paper). In such cases training should stop before this value of n. Why doesn't Fig. a) show this trend? Is there a neural network theorem or guideline stating what behaviour can be expected for different types of applications?
Similarly, in Fig. b) it seems that all errors continue to decrease. How should the optimal number of hidden nodes be selected given such a graph? Should I evaluate the number of hidden neurons at much higher values? (beyond $N_h$=98 I run into memory problems...).