I am relatively new to using XGBoost. Classification problems clearly don't get affected by the feature scaling as the new splits would take care of that. But when doing regression in XGBoost, aren't we 'fitting', say, a linear regression (objective "reg:linear" in xgboost) model at some step? If so, that step might fail if the feature attributes involved aren't of similar variance. Is it or is it not?
xgboost won't fit any linear trends to your data unless you specify
booster = "gblinear", which fits a small regression in the nodes. The
reg:linear objective tells it to use sum of squared error to inform its fit on a regression problem.
Linear regression with gradient boosted trees is unaffected by any feature scaling as long as the rank order of the feature doesn't change.
Yes, XGBoost (and in general decision trees) is invariant under features scaling (monotone transformations of individual ordered variables) if you set the
booster parameter to
gbtree (to tell XGBoost to use a decision tree model).
objective parameter refers to the function to be me minimised and not to the model.
'reg:linear' tells XGBoost to minimize the mean square error.
Sources: - https://onlinecourses.science.psu.edu/stat857/node/55 - https://homes.cs.washington.edu/~tqchen/pdf/BoostedTree.pdf (slide10, from the XGBoost author)