5
$\begingroup$

I have seen many posts online about tuning the random forest hyperparameters with a gridsearch. however, since the random forest creates trees with some randomness, does this have sense? Wouldn't the best parameters just be because of a lucky model?

Let me make an example. I tuned the parameters of an ExtraTreeClassifier, getting very nice results. However, training with the same parameters changing the random state (seed) of the model leads to terrible performance. Am I allowed to think that the model trained works well?

$\endgroup$
1
  • 3
    $\begingroup$ This is just one specific example of a very common question in statistics: "Given several groups of measurements, what's the probability that some statistic of those measurements is the same?" $\endgroup$
    – Sycorax
    Jun 28 at 13:37

2 Answers 2

3
$\begingroup$

You are right that randomness will play a role (like with many other algorithms including MCMC samplers for Bayesian models, XGBoost, LightGBM, neural networks etc.) in the results. The obvious way to minimize randomness in the results of any hyper-parameter optimization method for RF (whether it's random grid-search, grid search or some Bayesian hyperparameter optimization method) is to increase the number of trees (which reduces the randomness in the model behavior - albeit at the cost of an increased training time). Alternatively, you construct a surrogate model on top of the results that takes into account that the signal, of where the best model in the hyperparameter landscape is, is noisy through an appropriate amount of smoothing/regularization.

$\endgroup$
3
$\begingroup$

To add a little to @Björn's answer, when the model selection criterion is noisy (or there is a random element to the classifier) grid search (or random search) actually makes more sense than some more elegant or more efficient model selection procedures, such as gradient descent or Nelder-Mead simplex, where the randomness may affect the termination criterion for the optimisation algorithm (they generally stop when the improvement in performance or the "gradient" is small).

Randomness in the construction of a classifier is not ideal, so minimising it by e.g. using lots of trees is a good idea. One problem is that the noisyness of the classifier construction may make it easier to over-fit the model selection criteria if the "randomness" of a particular classifier just happens to suit the sampling variation in the validation set (or cross-validation) error.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.