Problem with building Optimizer using Random Forest as Surrogate Model

Question

I am trying to use Random Forest as Surrogate Model instead of Gaussian Process for my Bayesian Optimizer Framework and already studied the concept of it through SMAC and mlrMBO Papers. I use the Python package Pyrfr https://github.com/automl/random_forest_run to build Random Forest. But the prediction of the new data give me strange mean and variance value at the first few Observations of the Optimizer, it gives me constant mean and variance. As the Observations increase, the mean and variance of Random Forest start to change but still quite useless to use with the Infill Function for sampling next best Configuration. I use Expected Improvement(EI) as Infill Function(Acquisition Function in some cases). If i use the mean and variance from the Random Forest, EI will result in 0 because of constant value in mean and variance. The same thing happened when i used Random Forest from Scikit-learn and using this Package here to get the variance (The jackknife method) http://contrib.scikit-learn.org/forest-confidence-interval/index.html

Furthermore i could use it wrong and the don't get the result that i want. I tested it on a synthetic Function , in this case the famous Rosebrock function. If i optimize this Function using normal Gaussian Process, i could reach global minimum in less than 30 Iterations, compare to Random Forest , i have to use the first 10-15 iterations to sample enough values for the Infill Function to show different values, otherwise it will be constant or 0. And even after you have enough evaluations for RF, picking the next best Configurations is not that good like in GP, i couldn't reach global minimum with RF.

So the question is, could anybody give me advices how to use RF correctly or could show what i did wrong here ? I will provide more details if you have question or don't understand what i mean here. Thanks !

Answer 1

Random forests (either for regression or for classification) create piecewise constant functions as their predictions. I'm pretty sure this is the ultimate source of your problem. Some speculation:

• Whatever optimizer you're using to find the next point to visit on the surrogate surface either can't easily find the maximum. This depends on what your surrogate surface optimizer is; anything with a gradient computation will struggle here.

• The random forest predictions are too coarse, and is only very poorly modeling the underlying function. This is where using a standard global optimization test problem can help. Try tracking the average residual error for a model that uses the random forest surrogate and a model that uses a Gaussian process surrogate. I think you'll find that the error is lower for the GP.