4
$\begingroup$

This is a specification or continuation of my prior question:

[Is it 'fair' to set a seed in a random forest regression to yield the highest accuracy?

Its valid to use cross-validation to set parameters. And we can use SKL's GridSearch method, which uses k-fold cross-validation to train hyperparameters. But consensus seems to be that it's not valid to use cross-validation to find the best-performing random_state = seed

Why?

$\endgroup$
2
  • 2
    $\begingroup$ The non-deterministic nature of ML algorithms is their strength, not their weakness. Imagine trying to find the global minima in a plane. By tuning the hyper-parameters you are changing the means of navigation through that plane. By changing seeds you are just selecting a different starting point. If the model had a different plane, the first way (changing the means of navigation) might transfer favorably and help you find a minima there. The second way probably won't. However I might be wrong, It'll be interesting to see other peoples opinions on this matter so I won't answer this one. $\endgroup$ – Djib2011 Apr 19 '18 at 23:37
  • $\begingroup$ I totally agree with what @jld wrote in a separate question (StackExchange does not allow me to add a comment). I wrote a separate answer in your initial question, in which I explain why it is both valid and correct to set the seed using CV or any type of Model Validation. $\endgroup$ – Ioannis Kourouklides May 7 '20 at 11:27
5
$\begingroup$

Provided you do your cross-validation properly (i.e. cross validate the whole procedure) I don't think it's "wrong", I think the most likely result is just that it won't be helpful. Doing this CV properly though means you have to be careful to include these decisions in each fold. It's common to see people forgetting to include modeling decisions in the cross validation and thereby biasing their assessments. I'm going to use the term procedure rather than model for this combination process just to not confuse the two.

To that end, consider the evaluation of two modeling procedures:

  1. within each fold of our CV we pick a random random seed and fit our random forest.

  2. within each fold of our CV we fit a RF for seed $s = 1, \dots, 100$ and take the best one.

In this case the first procedure is the combination of picking a seed then fitting the RF. The second procedure is the combination of fitting 100 RFs and picking the best one. In both cases we're properly CV-ing the model so we'll get a fair assessment of it.

I think it's fine to compare these so long as you think there is a possibility that different seeds might help. If you really believe there's no way this could help then you're only opening yourself up to type I errors by doing this. But there is a precedent for this sort of thing. In nonconvex optimization you generally only get a local optimum so it's common to do many random restarts and then pick the best of the collection of resulting local optima; see e.g. this paper How many random restarts are enough?: the authors aren't questioning if any random restarts should be done but rather how many. So in our case if we think there is a possible benefit to it then we can include it as a hyperparameter, but again we need to be explicit about how it's now part of the modeling procedure and so it needs to be cross validated.

I bet if you do this you'll find that, provided you're using enough trees, that the two will have indistinguishable CV values. In that case you ought to take the simpler model, which will be the one without the seed optimizing. But that's fine: you've compared two models correctly via cross validation and you picked the better one. We do that every day.

If anyone disagrees please let me know!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.