I have a random forest regression built using skl and I note that I yield different results based on setting the random seed to different values.
If I use LOOCV to establish which seed works best, is this a valid method?
$\begingroup$
$\endgroup$
5
-
2$\begingroup$ People do that in competitions. Though for academia that would be hard to justify. $\endgroup$– FirebugApr 19, 2018 at 22:20
-
2$\begingroup$ Think of an extreme case scenario: We play a game: we roll two dices and the ones of us who gets the higher sum wins. But actually I am allowed to roll the dice twice. Is that fair you? Let me point that setting the random seeds is an integral part of reproducible research and should always be done. That doesn't mean though that we should try many different seeds until we find a "favourable seed". $\endgroup$– usεr11852Apr 19, 2018 at 22:49
-
$\begingroup$ @usεr11852 what do you think of my comment on the currently accepted answer? I’m not sure if this is any different than random restarts like with kmeans. No one thinks we should be forced to accept the first run of it that we do, to the point that random restarts are built in to the standard function in R. Unless maybe you consider the model to be running kmeans 100 times rather than the model being just the single best clustering $\endgroup$– jldApr 19, 2018 at 23:07
-
$\begingroup$ No—it seems like the very definition of overfitting on actual randomness. $\endgroup$– Mark WhiteApr 20, 2018 at 3:33
-
$\begingroup$ @Chaconne: I fully support your point about the need for proper validation. That said I think there is a core difference in the two use-cases: In the case of k-means (or stochastic optimisation in general) we look for an "optimal set" of parameters while for CV we care for a "representative set". In the early case we strive to show "how good can we be" while in the later case "how good will we be". $\endgroup$– usεr11852Apr 20, 2018 at 20:46
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
10
The answer is no.
Your model gives a different result for each seed you use. This is a result of the non-deterministic nature of the model. By choosing a specific seed that maximizes the performance on the validation set means that you chose the "arrangement" that best fits this set. However, this does not guarantee that the model with this seed would perform better on a separate test set. This simply means that you have overfit the model on the validation set.
This effect is the reason you see many people that rank high in competitions (e.g. kaggle) on the public test set, fall way off on the hidden test set. This approach is not considered by any means the correct approach.
Edit (not directly correlated to the answer, but I found it interesting)
You can find an interesting study showing the influence of random seeds in computer vision here. The authors first prove that you can achieve better results when using a better seed than the other and offer the critique that many of the supposed SOTA solutions could be merely better seed selection than the others. This is described in the same context as if it is cheating, which in all fairness it kind of is... Better seed selection does not make your model inherently better, it just makes it appear better on the specific test set.
-
6$\begingroup$ Yes, this is why cross-validation is such a strong technique and also why people use both a validation and a test set (one to base the model selection on and one to get an unbiased evaluation). $\endgroup$– Djib2011Apr 19, 2018 at 22:49
-
1$\begingroup$ I’m not convinced. With nonconvex optimization it’s routine to do random restarts because different seeds can lead to very different model parameter estimates and just by bad luck you can get bad convergence. Eg with kmeans this is well known. With the random forest just by chance maybe your model ends up making too many subpar splits. I don’t think it’s fitting noise to recognize that different runs lead to different model parameter estimates and some may actually generalize better than others. This is all conditioned upon properly assessing out of sample performance, of course. $\endgroup$– jldApr 19, 2018 at 23:01
-
5$\begingroup$ @user2723494 It is valid for hyper-parameters in hopes of increasing performance without the cost of generalization. However fine-tuning the parameters again and again on the validation set would produce the same effect I described (overfitting on the validation set). Because random seeding is by its nature stochastic it is far more likely to improve the performance due to overfitting than due to have actually produced a better model. $\endgroup$– Djib2011Apr 19, 2018 at 23:13
-
5$\begingroup$ @Chaconne I didn't say that was impossible. However by this procedure it is far more likely to select a seed that overfits than one that actually produces a better model... $\endgroup$– Djib2011Apr 19, 2018 at 23:15
-
5$\begingroup$ To be honest, I've never considered seeding ML algorithms (and depriving them from their stochastic nature) a good practice. The OP created a new question asking just this. I'd be interested in reading your response! $\endgroup$– Djib2011Apr 19, 2018 at 23:43
$\begingroup$
$\endgroup$
4
For some algorithms a bad initialization may matter and may be due to the particular random seed. In such cases, it may make sense to try to find a good initialitzation (=good random seed) that then leads to a good performance (or to find a way of modifying the training to reduce such effects). However, one should really be convinced that this is going on, because what we don't want to do - as others already pointed out - is to overfit our validation set by finding a seed that happens to produce a good result due to some ill-understood combination of the noisiness of the training process and the characteristics of the validation set (or sets in cross-validation).
In the particular case of the random forest algorithm, I don't think we are in a case where we want to optimize the seed, at all. What we can do instead is to increase the number of trees until the results no longer depend on the seed in any meaningful way. More trees don't lead to overfitting for RF (unlike for, say, XGBoost, for which the corresponding remedy would be to fit the model multiple times and average the predictions), more trees just takes random noise out of the validaiton set performance (and up to an extent improve performance). For RF, I'd argue such randomness is just "bad" in the sense that it obscures the best hyperparameters with noise and might be due to some chance combination of factors between training process & validation set characteristics, but we have no reason to think these fluctuations would reliably turn up on new data (such as an unseen test set). So, it makes sense to eliminate it as much as possible (to the degree that that's possible in terms of our computational budget for training and inference).
-
$\begingroup$ Do you agree with tuning DNNs on the validation set and reporting the test set measures? $\endgroup$– keramatOct 25, 2021 at 15:37
-
$\begingroup$ Not sure why you are asking, but in principle, yes. Of course, always trying to avoid things that are just noise mining. $\endgroup$– BjörnOct 25, 2021 at 15:55
-
$\begingroup$ I am speaking about tuning seed on the validation set. I am not sure that it means noise. I think starting weights from a good but randomly point can lead to a better local minima. $\endgroup$– keramatOct 26, 2021 at 10:15
-
1$\begingroup$ That's the big question, if this makes you end up somewhere truly better, then it's desirable, while if it does not and it's just overfitting to the validation set with a bit of extra randomness, then it's at best a waste of time and resources. $\endgroup$– BjörnOct 27, 2021 at 12:15
$\begingroup$
$\endgroup$
5
The short answer is YES, it is both fair and correct, contrary to what @Djib2011 wrote in a separate answer.
If you follow the usual procedure in ML, then setting the seed in this context does NOT lead to overfitting, contrary to the other answer here is falsely suggesting. You can call it "seed optimization" or "seed hacking", but definitely not overfitting.
Also, YES, using any type of Cross-Validation (including LOOCV) is acceptable, valid and correct. And you should use Model Validation actually (either a type of CV or something else).
Essentially, it is totally correct to treat the seed as a hyperparameter in this specific context.
It is actually accepted in both industry and academia (and competitions). There are published peer-reviewed papers describing this very procedure. Here are two good examples:
Also, it is very common in Unsupervised Learning, e.g. using k-means++ algorithm to set the seed for k-means algorithm. So, I do not understand the confusion of @Djib2011 or other people.
@jld in a separate answer goes in more depth to explain why this is not wrong and how to ensure you follow the correct procedure if you opt for performing CV as Model Validation. As it is explained, setting the set might or might not be useful, but this is an other story.
Caution:
There are at least six sources of randomness in ML (that can be set using a seed). You just described one of them. In some of those other contexts, it is wrong to set the seed. Again, this is an other story. Six of those sources are described below:
-
2$\begingroup$ What is suggested by OP is to find a seed that produces a higher estimation of the performance. That is different from checking estimation stability across CV repetitions, and different to model averaging across seeds (as suggested in the Bengio article you cited). Variance in the estimated values is to be expected, as CV is often not sufficient to stabilize the estimation (see for instance Nadeau & Bengio, 2003). Therefore reseeding as OP suggested is more like reporting the upper bound of a confidence interval as if it was a mean. $\endgroup$ Jun 29, 2021 at 20:24
-
$\begingroup$ Let's agree to disagree. I completely disagree with you. More specifically: (1) I believe my answer (either you agree with what I wrote or not) does indeed answer the question of the OP. (2) "Therefore reseeding as OP suggested is more like reporting the upper bound of a confidence interval as if it was a mean." -> This does not make sense to me at all. As I explained, it is totally correct to treat the seed as a hyperparameter in this specific context. $\endgroup$ Apr 3, 2022 at 1:41
-
$\begingroup$ I am not sure what I meant myself so I'll start from scratch. Let's assume a classification procedure with a stochastic element S that can be manipulated by changing the seed. Suppose I repeat the procedure many times, changing S, keeping all accuracy values. My claim is that as far as the seed is concerned, the true accuracy (that would generalize to new data from the same population) is somewhere in the resulting interval. From what I understand your claim is that the true accuracy is the max value obtained from that sample. See also arxiv.org/abs/2104.00673 $\endgroup$ Apr 4, 2022 at 13:25
-
$\begingroup$ See also this paper, especially section 5: proceedings.mlsys.org/paper/2021/file/… $\endgroup$ Apr 4, 2022 at 15:08
-
$\begingroup$ I saw the whole paper. To my understanding, there is not anything contradictory between the aforementioned paper and my initial answer. Anyway, there was another question by the OP that the selected answer was compatible with my answer. $\endgroup$ Sep 14, 2022 at 7:53