I have a random forest regression built using skl and I note that I yield different results based on setting the random seed to different values.
If I use LOOCV to establish which seed works best, is this a valid method?
$\begingroup$
$\endgroup$
-
1$\begingroup$ People do that in competitions. Though for academia that would be hard to justify. $\endgroup$ – Firebug Apr 19 '18 at 22:20
-
2$\begingroup$ Think of an extreme case scenario: We play a game: we roll two dices and the ones of us who gets the higher sum wins. But actually I am allowed to roll the dice twice. Is that fair you? Let me point that setting the random seeds is an integral part of reproducible research and should always be done. That doesn't mean though that we should try many different seeds until we find a "favourable seed". $\endgroup$ – usεr11852 Apr 19 '18 at 22:49
-
$\begingroup$ @usεr11852 what do you think of my comment on the currently accepted answer? I’m not sure if this is any different than random restarts like with kmeans. No one thinks we should be forced to accept the first run of it that we do, to the point that random restarts are built in to the standard function in R. Unless maybe you consider the model to be running kmeans 100 times rather than the model being just the single best clustering $\endgroup$ – jld Apr 19 '18 at 23:07
-
$\begingroup$ No—it seems like the very definition of overfitting on actual randomness. $\endgroup$ – Mark White Apr 20 '18 at 3:33
-
$\begingroup$ @Chaconne: I fully support your point about the need for proper validation. That said I think there is a core difference in the two use-cases: In the case of k-means (or stochastic optimisation in general) we look for an "optimal set" of parameters while for CV we care for a "representative set". In the early case we strive to show "how good can we be" while in the later case "how good will we be". $\endgroup$ – usεr11852 Apr 20 '18 at 20:46
$\begingroup$
$\endgroup$
The answer is no.
Your model gives a different result for each seed you use. This is a result of the non-deterministic nature of the model. By choosing a specific seed that maximizes the performance on the validation set means that you chose the "arrangement" that best fits this set. However, this does not guarantee that the model with this seed would perform better on a separate test set. This simply means that you have overfit the model on the validation set.
This effect is the reason you see many people that rank high in competitions (e.g. kaggle) on the public test set, fall way off on the hidden test set. This approach is not considered by any means the correct approach.
-
5$\begingroup$ Yes, this is why cross-validation is such a strong technique and also why people use both a validation and a test set (one to base the model selection on and one to get an unbiased evaluation). $\endgroup$ – Djib2011 Apr 19 '18 at 22:49
-
1$\begingroup$ I’m not convinced. With nonconvex optimization it’s routine to do random restarts because different seeds can lead to very different model parameter estimates and just by bad luck you can get bad convergence. Eg with kmeans this is well known. With the random forest just by chance maybe your model ends up making too many subpar splits. I don’t think it’s fitting noise to recognize that different runs lead to different model parameter estimates and some may actually generalize better than others. This is all conditioned upon properly assessing out of sample performance, of course. $\endgroup$ – jld Apr 19 '18 at 23:01
-
5$\begingroup$ @user2723494 It is valid for hyper-parameters in hopes of increasing performance without the cost of generalization. However fine-tuning the parameters again and again on the validation set would produce the same effect I described (overfitting on the validation set). Because random seeding is by its nature stochastic it is far more likely to improve the performance due to overfitting than due to have actually produced a better model. $\endgroup$ – Djib2011 Apr 19 '18 at 23:13
-
4$\begingroup$ @Chaconne I didn't say that was impossible. However by this procedure it is far more likely to select a seed that overfits than one that actually produces a better model... $\endgroup$ – Djib2011 Apr 19 '18 at 23:15
-
4$\begingroup$ To be honest, I've never considered seeding ML algorithms (and depriving them from their stochastic nature) a good practice. The OP created a new question asking just this. I'd be interested in reading your response! $\endgroup$ – Djib2011 Apr 19 '18 at 23:43
$\begingroup$
$\endgroup$
The short answer is YES, it is both fair and correct, contrary to what @Djib2011 wrote in a separate answer.
If you follow the usual procedure in ML, then setting the seed in this context does NOT lead to overfitting, contrary to the other answer here is falsely suggesting. You can call it "seed optimization" or "seed hacking", but definitely not overfitting.
Also, YES, using any type of Cross-Validation (including LOOCV) is acceptable, valid and correct. And you should use Model Validation actually (either a type of CV or something else).
Essentially, it is totally correct to treat the seed as a hyperparameter in this specific context.
It is actually accepted in both industry and academia (and competitions). There are published peer-reviewed papers describing this very procedure. Here are two good examples:
Also, it is very common in Unsupervised Learning, e.g. using k-means++ algorithm to set the seed for k-means algorithm. So, I do not understand the confusion of @Djib2011 or other people.
@jld in a separate answer goes in more depth to explain why this is not wrong and how to ensure you follow the correct procedure if you opt for performing CV as Model Validation. As it is explained, setting the set might or might not be useful, but this is an other story.
Caution:
There are at least six sources of randomness in ML (that can be set using a seed). You just described one of them. In some of those other contexts, it is wrong to set the seed. Again, this is an other story. Six of those sources are described below:
