I am implementing an univariate time-series forecasting neural network that takes a sliding time window as input variables. I am using a feed-forward network with one hidden layer. Evaluation metric is RMSE. Data is divided into train, validation and test. Network weights are always randomly initialized.
I would like to modify and find the best value for certain parameter, for example the hidden layer neuron amount. For this example, I run the neural network with different hidden neurons (e.g. 1, 3, 5...) and compare the results in terms of the test metric.
(So far, no problem).
To have more statistical relevance, I run each experiment (e.g. hidden neurons=1) a number of times (e.g. 30 times) and observe the RMSE mean and standard deviation from those runs.
But in class, a professor suggested that another comparison method could be used: Run each experiment (e.g. hidden neurons=1) a number of times (e.g. 30 times), and pick the smaller RMSE from there. Then I would compare different configurations (e.g. hidden neurons=1, 3, 5,...) by observing the smallest RMSE from each case. Reason for this would be that picking the smallest RMSE from several runs might provide a closer value to the real global minimum from the training process.
Which of the two approaches is statistically more significant to evaluate the neural network performance and generalization? Are both correct?