I have a random forest regression built using skl and I note that I yield different results based on setting the random seed to different values.
If I use LOOCV to establish which seed works best, is this a valid method?
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
The answer is no.
Your model gives a different result for each seed you use. This is a result of the non-deterministic nature of the model. By choosing a specific seed that maximizes the performance on the validation set means that you chose the "arrangement" that best fits this set. However, this does not guarantee that the model with this seed would perform better on a separate test set. This simply means that you have overfit the model on the validation set.
This effect is the reason you see many people that rank high in competitions (e.g. kaggle) on the public test set, fall way off on the hidden test set. This approach is not considered by any means the correct approach.
Edit (not directly correlated to the answer, but I found it interesting)
You can find an interesting study showing the influence of random seeds in computer vision here. The authors first prove that you can achieve better results when using a better seed than the other and offer the critique that many of the supposed SOTA solutions could be merely better seed selection than the others. This is described in the same context as if it is cheating, which in all fairness it kind of is... Better seed selection does not make your model inherently better, it just makes it appear better on the specific test set.
For some algorithms a bad initialization may matter and may be due to the particular random seed. In such cases, it may make sense to try to find a good initialitzation (=good random seed) that then leads to a good performance (or to find a way of modifying the training to reduce such effects). However, one should really be convinced that this is going on, because what we don't want to do - as others already pointed out - is to overfit our validation set by finding a seed that happens to produce a good result due to some ill-understood combination of the noisiness of the training process and the characteristics of the validation set (or sets in cross-validation).
In the particular case of the random forest algorithm, I don't think we are in a case where we want to optimize the seed, at all. What we can do instead is to increase the number of trees until the results no longer depend on the seed in any meaningful way. More trees don't lead to overfitting for RF (unlike for, say, XGBoost, for which the corresponding remedy would be to fit the model multiple times and average the predictions), more trees just takes random noise out of the validaiton set performance (and up to an extent improve performance). For RF, I'd argue such randomness is just "bad" in the sense that it obscures the best hyperparameters with noise and might be due to some chance combination of factors between training process & validation set characteristics, but we have no reason to think these fluctuations would reliably turn up on new data (such as an unseen test set). So, it makes sense to eliminate it as much as possible (to the degree that that's possible in terms of our computational budget for training and inference).
The short answer is YES, it is both fair and correct, contrary to what @Djib2011 wrote in a separate answer.
If you follow the usual procedure in ML, then setting the seed in this context does NOT lead to overfitting, contrary to the other answer here is falsely suggesting. You can call it "seed optimization" or "seed hacking", but definitely not overfitting.
Also, YES, using any type of Cross-Validation (including LOOCV) is acceptable, valid and correct. And you should use Model Validation actually (either a type of CV or something else).
Essentially, it is totally correct to treat the seed as a hyperparameter in this specific context.
It is actually accepted in both industry and academia (and competitions). There are published peer-reviewed papers describing this very procedure. Here are two good examples:
Also, it is very common in Unsupervised Learning, e.g. using k-means++ algorithm to set the seed for k-means algorithm. So, I do not understand the confusion of @Djib2011 or other people.
@jld in a separate answer goes in more depth to explain why this is not wrong and how to ensure you follow the correct procedure if you opt for performing CV as Model Validation. As it is explained, setting the set might or might not be useful, but this is an other story.
There are at least six sources of randomness in ML (that can be set using a seed). You just described one of them. In some of those other contexts, it is wrong to set the seed. Again, this is an other story. Six of those sources are described below: