Why is regression with Gradient Tree Boosting sometimes impacted by normalization (or scaling)?

Question

I read that normalization is not required when using gradient tree boosting (see e.g. https://stackoverflow.com/q/43359169/1551810 and https://github.com/dmlc/xgboost/issues/357).

And I think I understand that in principle there is no need for normalization when boosting regression trees.

Nevertheless, using xgboost for regression trees, I see that scaling the target can have a significant impact on the (in-sample) error of the prediction result. What is the reason for this?

Example for the Boston Housing dataset:

import numpy as np
import pandas as pd
import xgboost as xgb
from sklearn.metrics import mean_squared_error
from sklearn.datasets import load_boston

boston = load_boston()
y = boston['target']
X = boston['data']

scales = pd.Index(np.logspace(-6, 6), name='scale')
data = {'reg:linear': [], 'reg:gamma': []}
for objective in ['reg:linear', 'reg:gamma']:
    for scale in scales:
        xgb_model = xgb.XGBRegressor(objective=objective).fit(X, y / scale)
        y_predicted = xgb_model.predict(X) * scale
        data[objective].append(mean_squared_error(y, y_predicted))

pd.DataFrame(data, index=scales).plot(loglog=True, grid=True).set(ylabel='MSE')

Answer 1

A big part of the answer seems to be found in https://github.com/dmlc/xgboost/issues/799#issuecomment-181768076.

By default, base_score is set to 0.5 and this seems a bad choice for regression problems. When the average of the target is much higher or lower than base_score, the first x trees are just trying to catch the average, and less trees are left to solve the real task.

The solution thus seems simple: adjust base_score to the mean of the target to avoid impact from its scale on the regression result.

Especially for objective 'reg:gamma' this indeed seems to be the clue, whereas for 'reg:linear' it provides only a partial improvement:

data = {'reg:linear': [], 'reg:gamma': [], 'reg:linear - base_score': [], 'reg:gamma - base_score': []}
for objective in ['reg:linear', 'reg:gamma']:
    for scale in scales:
        xgb_model = xgb.XGBRegressor(objective=objective).fit(X, y / scale)
        y_predicted = xgb_model.predict(X) * scale
        data[objective].append(mean_squared_error(y, y_predicted))

for objective in ['reg:linear', 'reg:gamma']:
    for scale in scales:
        base_score = (y / scale).mean()
        xgb_model = xgb.XGBRegressor(objective=objective, base_score=base_score).fit(X, y / scale)
        y_predicted = xgb_model.predict(X) * scale
        data[objective + ' - base_score'].append(mean_squared_error(y, y_predicted))

styles = ['g-', 'r-', 'g--', 'r--']
pd.DataFrame(data, index=scales).plot(loglog=True, grid=True, style=styles).set(ylabel='MSE')

So the remaining question reduces to: Why is there still sometimes an impact of scaling the target with objective 'reg:linear', even after adjusting base_score to the mean of the (scaled) target?

Answer 2

Trees really do not depend on scaling, and a linear boosting algorithm shouldn't either.

However, we have to keep in mind XGBoost is a Gradient Boosting algorithm that tries to optimize a loss function based on the addition of models, through gradient descent.

Why is this important? Gradient descent update rules operate based on the estimation of an optimal direction for updating plus a step length for a step in this direction. The length of this step isn't obvious at all, and in XGBoost it's estimated based on a second-order expansion of the objective function gradient .

I have to investigate it further, but I guess the reason might lie on this optimization.