2
$\begingroup$

I have fitted several neural networks to some training data using different parameter settings for weight decay, nodes, max iterations etc. As I dealt with time series data I chose a form of rolling-window cross validation similar to here where accuracy indicated what percentage of test data instances where classified correctly. Naturally I would select the parameter settings which produced the highest accuracy for my final model.

However I noticed that it became quite tricky to reproduce my results, despite taking the same input parameters. So as I rerun the estimation process with one set of parameters the network found different local minima resulting in different accuracies.

Now I could do just that and than point to the estimated model which produced the highest accuracy as the "best". Is this approach valid however?

Or is it not "misusing" the test-data because now you are implicitly fitting the model to that data? Do you believe that a model fitted in such a way will produce better results on new data than a model with a lower accuracy score?

How do you go about in making neural networks "more stable"?

$\endgroup$
1
$\begingroup$

Now I could do just that and than point to the estimated model which produced the highest accuracy as the "best". Is this approach valid however?

You cannot use the test set to choose your model. You could select the model which produced the highest accuracy on the validation set.

How do you go about in making neural networks "more stable"?

If you don't trust much your validation set to reflect the test set, you could train several neural networks and combine their outputs using some ensemble methods.

$\endgroup$
1
  • 1
    $\begingroup$ I didn't leave a test set, instead I divided the data in a training set and produced rolling forecast of many neural networks with different parameters for decay etc for the cv set. The parameters resulting in the highest accuracy on the cv set were picked as "best". However now my problem is, that I could reestimate the weights of the model with these same parameters in order to improve the already high accuracy on the cv set - however this feels like cheating/overfitting. $\endgroup$ – RubenStrenzke Sep 22 '16 at 15:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.