I have fitted several neural networks to some training data using different parameter settings for weight decay, nodes, max iterations etc. As I dealt with time series data I chose a form of rolling-window cross validation similar to here where accuracy indicated what percentage of test data instances where classified correctly. Naturally I would select the parameter settings which produced the highest accuracy for my final model.
However I noticed that it became quite tricky to reproduce my results, despite taking the same input parameters. So as I rerun the estimation process with one set of parameters the network found different local minima resulting in different accuracies.
Now I could do just that and than point to the estimated model which produced the highest accuracy as the "best". Is this approach valid however?
Or is it not "misusing" the test-data because now you are implicitly fitting the model to that data? Do you believe that a model fitted in such a way will produce better results on new data than a model with a lower accuracy score?
How do you go about in making neural networks "more stable"?