3
$\begingroup$

I was running an experiment with neural networks and I could see that training the neural network model with the same input give different results on the test. This should be due to the different weight initializations. The question is how to choose a model from these iterations. Do we need to take the average of results or the best result or the take the average of weights and then retrain and get a single result?

$\endgroup$
1
$\begingroup$

Normally you'd take the best result.

However you run a risk of overfitting on the test set. You might find a set of initial weights that cause the model, when trained, to perform the best on that particular test set, which might not generalize well on different test sets.

I'd suggest having two sets for this purpose, a validation set where you'd perform the initial experiments as you did. Then test the best of these networks on the test set and see if it achieves the same performance as it did on the validation set.

$\endgroup$
  • $\begingroup$ yes a validation data is used, actually the training data used is split again to train and validation data, and the performance on this data is good. $\endgroup$ – prashanth Aug 26 at 6:20
  • $\begingroup$ Then you'd take the best out of those models, just use the test set only once (for the best model to evaluate it). Don't use the test set for model selection $\endgroup$ – Djib2011 Aug 26 at 8:18
  • $\begingroup$ @ Djib2011 okay. you mean to say that select the best model (that is trained using the same input data but different initializations) that gives the best results on the test set, right? $\endgroup$ – prashanth Aug 26 at 9:00
  • $\begingroup$ No I mean that the test set shouldn't be used for model selection. So you can train $N$ models on the training set using the same input data and different initializations. Evaluate those $N$ models on the validation set in order to determine the best one and then re-evaluate that best one on the test set to see if it is actually good. $\endgroup$ – Djib2011 Aug 26 at 12:41
  • $\begingroup$ okay. makes sense $\endgroup$ – prashanth Aug 27 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.