(I am mostly self-taught in this area, and apologize if there is a simple solution that I managed to miss somewhere along the way.)
For several of my models (applied to different datasets), I have noticed that the seed used in the model fitting process can have a significant impact on the results. I understand that we should expect some variability with the results in machine learning; however, the variability I was seeing was enough to change the interpretation of the results--not something that fills me with confidence in my models. I considered taking a random sample of random seeds and taking the average of the coefficients produced, but that would only work for models with coefficients. I considered taking that approach and averaging the chosen hyper-parameters, but the resulting model would still be prone to seed variance. I had the thought that I should include in the cross-validation process a means of minimizing variance in parameters, hyper-parameters or fitted valued across random seeds, but I did not want to experiment with applying that step without being able to justify that it could produce more robust results.
I have seen this topic broached on other posts, but the prescriptions have included everything from add features, remove features, use more folds, use less folds, use a different error function, use a different model, etc. I wanted to pose this specific question to get more precise advice for how to address this problem.
set.seed(5678); rbinom(1, 100, .3)
returns $24$ andset.seed(1234); rbinom(1, 100, .3)
returns $28.$ Certainly, noticeably different, but not significantly different:prop.test(c(24, 28), c(100,100))$p.val
returns $0.628656;$ far above $5\% = 0.05.$ $\endgroup$