In machine learning - notably ensemble methods such as random forest, gradient boosting, extreme gradient boosting etc - can we say that the effect obtained for one predictor is ADJUSTED for all other predictors?

Details: Let's say we have a classicial linear regression model, like so:

Y = X1 + X2 + Xn ...

In this case, we could say that the coefficient for X1 is adjusted for the effect of X2, Xn etc. Hence, we obtain adjusted estimates of the effect of each predictor.

But is that also true for random forest, gradient boosting, extreme gradient boosting etc? Those tree-based models provide several important parameters (variable importance, partial dependence plots etc). But is the effect of each predictor (for example, the association seen in partial dependence plots) adjusted for all other predictors? Can we really say that?

  • 2
    $\begingroup$ I would say in general, no. In OLS, each variable has a chance to influence the model, while in tree-based methods, a variable may be excluded from the model entirely. These excluded variables are not necessarily the same as having a coefficient of 0 in the linear model. For example, in the presence of a dominant variable, a less important variable may not be included in the tree even if it is important to some subset of the observations (ie, an interaction effect). A properly specified linear model would pick up that relationship. $\endgroup$ – lmo Jun 9 at 11:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.