4
$\begingroup$

I am relatively new to using XGBoost. Classification problems clearly don't get affected by the feature scaling as the new splits would take care of that. But when doing regression in XGBoost, aren't we 'fitting', say, a linear regression (objective "reg:linear" in xgboost) model at some step? If so, that step might fail if the feature attributes involved aren't of similar variance. Is it or is it not?

$\endgroup$
1
  • $\begingroup$ If there is something I am missing about XGBoost or in general regarding CARTs please help me by providing appropriate references! $\endgroup$ Sep 1 '16 at 13:09
4
$\begingroup$

xgboost won't fit any linear trends to your data unless you specify booster = "gblinear", which fits a small regression in the nodes. The reg:linear objective tells it to use sum of squared error to inform its fit on a regression problem.

Linear regression with gradient boosted trees is unaffected by any feature scaling as long as the rank order of the feature doesn't change.

$\endgroup$
1
1
$\begingroup$

Yes, XGBoost (and in general decision trees) is invariant under features scaling (monotone transformations of individual ordered variables) if you set the booster parameter to gbtree (to tell XGBoost to use a decision tree model).

The XGBoost objective parameter refers to the function to be me minimised and not to the model. 'reg:linear' tells XGBoost to minimize the mean square error.

Sources: - https://onlinecourses.science.psu.edu/stat857/node/55 - https://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf (slide10, from the XGBoost author)

$\endgroup$
1
  • 1
    $\begingroup$ It is root mean square error, not mean square error $\endgroup$
    – Anake
    May 30 '17 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.