How to correct for results varying with random seed?

Question

(I am mostly self-taught in this area, and apologize if there is a simple solution that I managed to miss somewhere along the way.)

For several of my models (applied to different datasets), I have noticed that the seed used in the model fitting process can have a significant impact on the results. I understand that we should expect some variability with the results in machine learning; however, the variability I was seeing was enough to change the interpretation of the results--not something that fills me with confidence in my models. I considered taking a random sample of random seeds and taking the average of the coefficients produced, but that would only work for models with coefficients. I considered taking that approach and averaging the chosen hyper-parameters, but the resulting model would still be prone to seed variance. I had the thought that I should include in the cross-validation process a means of minimizing variance in parameters, hyper-parameters or fitted valued across random seeds, but I did not want to experiment with applying that step without being able to justify that it could produce more robust results.

I have seen this topic broached on other posts, but the prescriptions have included everything from add features, remove features, use more folds, use less folds, use a different error function, use a different model, etc. I wanted to pose this specific question to get more precise advice for how to address this problem.

Answer 1

First of all, observing substantial variation depending on the random seed means that you should explore this in more detail, just like you'd explore and deal with any other important source of random uncertainty in your modeling process:

• At the very least, measure the variance you get from this source of variation.
  Do this by re-running the entire fitting procedure with varying random seeds sufficiently often to get an estimate of the variance this causes in your final results (e.g. various figures of merit of the verification/validation of the model).

Once you know the variance and know that it is important, there are usually 2 very different approaches to deal with random uncertainty:

• You can reduce the variance in the model training process, i.e. stabilize it/make it more robust. This is usually done by restricting model complexity, e.g. heavier regularization, using less features from the very beginning (data-driven feature selection will typically not help here, but rather exhibit unstable selection of features), put a penalty on hyperparameter optimization that acts towards less complex models, ...

• The 2nd approach is to average over a sufficiently large population that exhibits the variance in question, in order to obtain a "mean" that is subject to less variance.
  This would be an aggregated model consisting of an ensemble of models that were obtained from varying random seeds in your case.

---

The verification and validation of the final optimized model should not enter these thoughts in any way: that part needs to be entirely independent of the training process.

However, the final model validation should include ruggedness wrt. random seed.

---

More or less folds in the cross validation should ideally not influence the cross validation results. Looking at variance between folds without distinguishing variance due to finite number of tested cases (which varies per fold but it constant once you look at all folds of a CV run) from variance due to model instability will give a hard to interpret conglomerate of these two factors.

There are some optimization heuristics (like the one-sd-rule) that nevertheless use between-fold-variance. Since you do have substantial model instability, you'd need to dig down and at least find out roughly what the dominating source of variance is.