Feature subsampling with gradient boosting

Question

A key component in building random forest models is feature subsampling, i.e., building each individual tree with only a percentage of predictors chosen randomly by tree. The literature often suggests a “rule of thumb” that, if there are $p$ predictors, we use $\sim\sqrt{p}$ predictors per tree in classification models or $\sim p/3$ predictors per tree in regression models (see Hastie and Tibshirani's Elements of Statistical Learning, section 15.3, for example).

My question is this: if we build a gradient boosting model that includes feature subsampling (using a package such as XGBoost), should we still adhere to this “rule of thumb” of $\sqrt{p}$ or $p/3$? In other words, in a model that “learns” as it builds trees, is it still appropriate to restrict the number of predictors per tree to the same extent as in a random forest?

Answer 1

A large motivation in restricting the number of predictors available to each learner in a random forest is to encourage variance between the trees. Because each tree has the same starting point, tricks like row and column subsampling are necessary to ensure that you don't have the same tree multiple times. This isn't nearly as big a problem for boosting, where trees are built residually to each other. Each tree gets a new, adjusted starting point for which a new, different tree structure will be optimal.

Subsampling by rows and columns still increases variance between trees and allows your model to converge faster with boosting, but it is not essential. $p/3$ or $\sqrt p$ seems like it would be too low for most boosting problems. Interacted signal will be harder to find if a pair of variables have such a small chance of existing together in the same tree. I have seen occasional and marginal gains in predictive power at around $3/4$.