I have a random forest regression model with 1000 trees, having 16 parameters (using python scikit-learn). The estimator can predict a target value with cross validated r2 score of 0.87 +/- 0.03. I would like to find the global (or at least close to global) maximum of the random forest, in order to identify the best values of the 16 parameters that yield this optimum. What is the best algorithm to reach the maximum of a random forest estimator? Is there a python implementation of this algorithm?


It's easy with scipy.optimize.fmin, evaluating your model on sampled points or rolling your own optimization routine. This is the bread and butter of how many models work, so it's worth learning about in great detail. See for instance, these lecture notes.

This probably isn't what you want though. The $r^2$ you cross-validated is for the distribution of the input space. The $\operatorname*{arg\,max}_x f(x)$ that the random forest will evaluate to will probably be in a low confidence neighborhood, away from the support of the training data.

You want a model that generates confidence intervals. In Scikit-Learn, GaussianProcess and GradientBoostingRegressor both do this. Gaussian Processes are excellent for this problem if you don't have more than a thousand training observations.

If you can collect more data after consulting with your model, then you evaluate your black box at the $\arg\max$ of the upper confidence bound, add the result as a new data-point and repeat until there is no change.

This problem is known as the Contextual Bandit. The choice of confidence bound is dependent on the exploration until convergence/exploitation of the intermittent values that you want. Since you don't care about how poorly the model performs while training, you'd pick a large upper confidence bound so the model will converge faster.

There are several related toolkits for this sort of problem. Spearmint, Hyperopt, and MOE.

If you cannot collect more data then you should take the $\arg\max$ of the lower confidence bound of the model. This penalizes predictions that the model is uncertain of.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.