I'm a bit uncertain about the correct experimental procedure on when to set random seeds when training machine learning models with random components or initializations.
Let's say I had to create a supervised learning model for some classification task. To start, I would set a random seed for doing train/test splits, for cross-validation splits on the training set, and for model random states. This way all the results would be reproducible. Once that is done, I would take the best model from the cross-validation results and use that for the final model.
However, I am uncertain, when training the final model should I be setting the model's random state using the same random seed I used before? I find that in some cases when training the final model without setting the random state I get highly variable performance metrics on the test set.
Also, due to this issue should I have used several random seeds in the cross-validation phase and averaged the results or was the original approach fine?