Parallelizing GBM in R

Question

I'm able to parallelize randomForest based on the foreach function in the following way:

model_rf <- foreach(ntree=rep(250, 4), .combine=combine, .multicombine=TRUE, .packages='randomForest') %dopar%
    randomForest(x = as.matrix(x.train), y = y.train, ntree=ntree, mtry = floor(ncol(x.train)/3))

prediction_rf <- predict(model_rf, x.test)

But trying to do the same thing with gbm (like below) does not work:

model_gbm <- foreach(ntree=rep(1250, 4), .combine=combine, .multicombine=TRUE, .packages='gbm') %dopar%
    gbm.fit(x = as.matrix(x.train), y = y.train, distribution = "gaussian", n.trees = ntree, interaction.depth = 1, verbose = F)

prediction_gbm <- predict(model_gbm, x.test, n.trees = 5000)

This throws out the following error:

error calling combine function:
<simpleError in fun(result.1, result.2, result.3, result.4): Argument must be a list of randomForest objects>

What is the right way to combine and predict with gbm?

---

P.S. I also tried using the gbm function with n.cores=4 instead of the gbm.fit function, but I got the following error:

Error: protect(): protection stack overflow

Answer 1

GBM is an inherently sequential algorithm. Each iteration depends on the results of the previous iteration. Contrast this with Random Forest where the algorithm is embarrassingly parallel -- every tree is independent of each other. This allows RF to be parallelized in ways GBM cannot.

The combine function you are using is from the randomForest package and it is crafted to specifically work on randomForest objects, as the error message says.

In summary you are trying to parallelize an inherently sequential algorithm and then combine the results using a function meant to concatenate independent randomForests.

If you're looking for performance gains for GBM models, check out the xgboost package. Parts of GBM can be parallel, such as the construction of the decision trees. This is the approach xgboost takes.

Answer 2

As other people have pointed out, you cannot combine multiple gbm models as it is not an ensemble learning technique like randomForest.

However, the challenge in GBM is hyper parameter optimisation( finding the parameters that yield the best results ). This can parallelised easily with the help of big data system.