I have a random forest regression built using skl and I note that I yield different results based on setting the random seed to different values.
If I use LOOCV to establish which seed works best, is this a valid method?
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.Sign up to join this community
The answer is no.
Your model gives a different result for each seed you use. This is a result of the non-deterministic nature of the model. By choosing a specific seed that maximizes the performance on the validation set means that you chose the "arrangement" that best fits this set. However, this does not guarantee that the model with this seed would perform better on a separate test set. This simply means that you have overfit the model on the validation set.
This effect is the reason you see many people that rank high in competitions (e.g. kaggle) on the public test set, fall way off on the hidden test set. This approach is not considered by any means the correct approach.
The short answer is YES, it is both fair and correct, contrary to what @Djib2011 wrote in a separate answer.
If you follow the usual procedure in ML, then setting the seed in this context does NOT lead to overfitting, contrary to the other answer here is falsely suggesting. You can call it "seed optimization" or "seed hacking", but definitely not overfitting.
Also, YES, using any type of Cross-Validation (including LOOCV) is acceptable, valid and correct. And you should use Model Validation actually (either a type of CV or something else).
Essentially, it is totally correct to treat the seed as a hyperparameter in this specific context.
It is actually accepted in both industry and academia (and competitions). There are published peer-reviewed papers describing this very procedure. Here are two good examples:
Also, it is very common in Unsupervised Learning, e.g. using k-means++ algorithm to set the seed for k-means algorithm. So, I do not understand the confusion of @Djib2011 or other people.
@jld in a separate answer goes in more depth to explain why this is not wrong and how to ensure you follow the correct procedure if you opt for performing CV as Model Validation. As it is explained, setting the set might or might not be useful, but this is an other story.
There are at least six sources of randomness in ML (that can be set using a seed). You just described one of them. In some of those other contexts, it is wrong to set the seed. Again, this is an other story. Six of those sources are described below: