3
$\begingroup$

Do machine learning algorithms like Boosted Regression Trees (in the R package (gbm)) follow the same statistical assumptions of not including correlated predictor variables in GLM?

i.e. If I have two correlated predictors (rsq=.7) should I be including both into my BRT model?

Any input or thoughts on this question would be greatly appreciated.

$\endgroup$
7
  • 1
    $\begingroup$ Linear regression IS a machine learning algorithm! $\endgroup$
    – Zach
    Mar 2 '13 at 2:27
  • 1
    $\begingroup$ Zack, that still doesnt answer my question, are the statistical assumptions the same between GLM and BRTs? $\endgroup$
    – I Del Toro
    Mar 2 '13 at 2:36
  • 1
    $\begingroup$ Since when does regression assume multi-collinearity? $\endgroup$ Mar 2 '13 at 3:04
  • $\begingroup$ Dirk: Ive edited my question. $\endgroup$
    – I Del Toro
    Mar 2 '13 at 3:06
  • 2
    $\begingroup$ LMs and GLMs don't make any assumptions, statistical or otherwise, about the relationships among predictors. This is fortunate because pretty much the only time they'll be all neat and uncorrelated is for balanced experimental data. $\endgroup$ Mar 2 '13 at 18:45
1
$\begingroup$

Uh, oh, old question... !

Linear model

A linear regression model usually has one or more of three purposes:

  1. Effect estimation
  2. Testing hypotheses
  3. Prediction

While there is no multicollinearity assumption behind the classic normal linear model, strong multicollinearity is problematic regarding the first two items:

  • Effect estimates are unnatural due to Ceteris Paribus clause, so are corresponding hypothesis tests.
  • Multicollinearity blows up standard errors of estimated effects, which influences p values and confidence intervals accordingly.

Predictions are unaffected. So if bullet point 3) is the only purpose of your model, you can usually enjoy life even under strong multicollinearity.

Boosted trees

For boosted trees, it is quite the same:

  • If you are only after predictions, multicollinearity is no problem.

  • However, if you use your model to derive information on the effects (e.g. by studying partial dependence plots, any sort of variable importance, SHAP decompositions etc.), multicollinearity is similarly problematic as for linear regression.

  • You don't do hypothesis tests with boosted trees, so bullet point 2) is irrelevant here.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.