This is a specification or continuation of my prior question: Is it 'fair' to set a seed in a random forest regression to yield the highest accuracy?
Its valid to use cross-validation to set parameters. And we can use SKL's GridSearch method, which uses k-fold cross-validation to train hyperparameters.
But consensus seems to be that it's not valid to use cross-validation to find the best-performing random_state = seed.
Why?
Provided you do your cross-validation properly (i.e. cross validate the whole procedure) I don't think it's "wrong", I think the most likely result is just that it won't be helpful. Doing this CV properly though means you have to be careful to include these decisions in each fold. It's common to see people forgetting to include modeling decisions in the cross validation and thereby biasing their assessments. I'm going to use the term procedure rather than model for this combination process just to not confuse the two.
To that end, consider the evaluation of two modeling procedures:
within each fold of our CV we pick a random random seed and fit our random forest.
within each fold of our CV we fit a RF for seed $s = 1, \dots, 100$ and take the best one.
In this case the first procedure is the combination of picking a seed then fitting the RF. The second procedure is the combination of fitting 100 RFs and picking the best one. In both cases we're properly CV-ing the model so we'll get a fair assessment of it.
I think it's fine to compare these so long as you think there is a possibility that different seeds might help. If you really believe there's no way this could help then you're only opening yourself up to type I errors by doing this. But there is a precedent for this sort of thing. In nonconvex optimization you generally only get a local optimum so it's common to do many random restarts and then pick the best of the collection of resulting local optima; see e.g. this paper How many random restarts are enough?: the authors aren't questioning if any random restarts should be done but rather how many. So in our case if we think there is a possible benefit to it then we can include it as a hyperparameter, but again we need to be explicit about how it's now part of the modeling procedure and so it needs to be cross validated.
I bet if you do this you'll find that, provided you're using enough trees, that the two will have indistinguishable CV values. In that case you ought to take the simpler model, which will be the one without the seed optimizing. But that's fine: you've compared two models correctly via cross validation and you picked the better one. We do that every day.
If anyone disagrees please let me know!
