Neural Networks - Acceptable to choose best model after several trys?

Question

I have fitted several neural networks to some training data using different parameter settings for weight decay, nodes, max iterations etc. As I dealt with time series data I chose a form of rolling-window cross validation similar to here where accuracy indicated what percentage of test data instances where classified correctly. Naturally I would select the parameter settings which produced the highest accuracy for my final model.

However I noticed that it became quite tricky to reproduce my results, despite taking the same input parameters. So as I rerun the estimation process with one set of parameters the network found different local minima resulting in different accuracies.

Now I could do just that and than point to the estimated model which produced the highest accuracy as the "best". Is this approach valid however?

Or is it not "misusing" the test-data because now you are implicitly fitting the model to that data? Do you believe that a model fitted in such a way will produce better results on new data than a model with a lower accuracy score?

How do you go about in making neural networks "more stable"?

Answer 1

Now I could do just that and than point to the estimated model which produced the highest accuracy as the "best". Is this approach valid however?

You cannot use the test set to choose your model. You could select the model which produced the highest accuracy on the validation set.

How do you go about in making neural networks "more stable"?

If you don't trust much your validation set to reflect the test set, you could train several neural networks and combine their outputs using some ensemble methods.