3
$\begingroup$

I'm a bit uncertain about the correct experimental procedure on when to set random seeds when training machine learning models with random components or initializations.

Let's say I had to create a supervised learning model for some classification task. To start, I would set a random seed for doing train/test splits, for cross-validation splits on the training set, and for model random states. This way all the results would be reproducible. Once that is done, I would take the best model from the cross-validation results and use that for the final model.

However, I am uncertain, when training the final model should I be setting the model's random state using the same random seed I used before? I find that in some cases when training the final model without setting the random state I get highly variable performance metrics on the test set.

Also, due to this issue should I have used several random seeds in the cross-validation phase and averaged the results or was the original approach fine?

$\endgroup$

1 Answer 1

0
$\begingroup$

There’s a discussion if Is random state a parameter to tune? There are known results of non-reproducible results of neural networks or reinforcement learning algorithms when using different seeds. One of the answers to the above thread applies to your question as well:

… As a consequence, random state values which performed well in the validation set do not correspond to those which would perform well in a new, unseen test set. Indeed, depending on the algorithm, you might see completely different results by just changing the ordering of training samples.

Yes, results depend on the seed, but you have no guarantee that the seed that worked for the cross-validation splits would work equally well for model trained on the whole data.

$\endgroup$
4
  • $\begingroup$ Thanks! So just to confirm I should be using one preset random seed (not tuned) when initializing my neural network model in all experiments even the final training. $\endgroup$ Jul 6, 2021 at 17:26
  • $\begingroup$ @VinhyDahPooh you can, most people probably would use same seed, but this should not matter & not be your concern. $\endgroup$
    – Tim
    Jul 6, 2021 at 17:56
  • $\begingroup$ Awesome thanks! $\endgroup$ Jul 6, 2021 at 18:07
  • $\begingroup$ @VinhyDahPooh if it answered your question, you can mark it as solved. $\endgroup$
    – Tim
    Jul 6, 2021 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.