I am relatively new to using XGBoost. Classification problems clearly don't get affected by the feature scaling as the new splits would take care of that. But when doing regression in XGBoost, aren't we 'fitting', say, a linear regression (objective "reg:linear" in xgboost) model at some step? If so, that step might fail if the feature attributes involved aren't of similar variance. Is it or is it not?
$\begingroup$
$\endgroup$
1
asked Sep 1, 2016 at 13:06
-
$\begingroup$ If there is something I am missing about XGBoost or in general regarding CARTs please help me by providing appropriate references! $\endgroup$– Vineeth BhaskaraCommented Sep 1, 2016 at 13:09
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
xgboost won't fit any linear trends to your data unless you specify booster = "gblinear", which fits a small regression in the nodes. The reg:linear objective tells it to use sum of squared error to inform its fit on a regression problem.
Linear regression with gradient boosted trees is unaffected by any feature scaling as long as the rank order of the feature doesn't change.
-
1$\begingroup$ An attempted editor argues that, "It is not sum of squared error, it is root mean squared error. (See github.com/dmlc/xgboost/blob/master/src/objective/… )". $\endgroup$ Commented Oct 5, 2018 at 13:20
$\begingroup$
$\endgroup$
1
Yes, XGBoost (and in general decision trees) is invariant under features scaling (monotone transformations of individual ordered variables) if you set the booster parameter to gbtree (to tell XGBoost to use a decision tree model).
The XGBoost objective parameter refers to the function to be me minimised and not to the model. 'reg:linear' tells XGBoost to minimize the mean square error.
Sources:
-
1$\begingroup$ It is root mean square error, not mean square error $\endgroup$– AnakeCommented May 30, 2017 at 13:49