Statistically compare neural network hyperparameter values

Question

I am implementing an univariate time-series forecasting neural network that takes a sliding time window as input variables. I am using a feed-forward network with one hidden layer. Evaluation metric is RMSE. Data is divided into train, validation and test. Network weights are always randomly initialized.

I would like to modify and find the best value for certain parameter, for example the hidden layer neuron amount. For this example, I run the neural network with different hidden neurons (e.g. 1, 3, 5...) and compare the results in terms of the test metric.

(So far, no problem).

To have more statistical relevance, I run each experiment (e.g. hidden neurons=1) a number of times (e.g. 30 times) and observe the RMSE mean and standard deviation from those runs.

But in class, a professor suggested that another comparison method could be used: Run each experiment (e.g. hidden neurons=1) a number of times (e.g. 30 times), and pick the smaller RMSE from there. Then I would compare different configurations (e.g. hidden neurons=1, 3, 5,...) by observing the smallest RMSE from each case. Reason for this would be that picking the smallest RMSE from several runs might provide a closer value to the real global minimum from the training process.

Which of the two approaches is statistically more significant to evaluate the neural network performance and generalization? Are both correct?

Answer 1

Training NNs is a practical task where you are only interested in well-performing neural networks. A learning session that ends up in a poor local minimum is quickly discarded as the solution is irrelevant for practical use.

My advice is to train many neural networks with different hidden-node architectures. Pick the 3 best performing trained neural networks corresponding to each configuration, and compare their performance estimates across architectures. You should use your testset for this, while keeping the validation set aside.

You can compare regression neural networks using different metrics.