Please correct my assumption on how regression trees work

Question

I'm trying to understand how regression trees work, I've been experimenting with catboost and xgboost in python, and I'm getting results which I don't expect, can someone please clarify (and apologies in advance if this is a coding error)

I've generated test data by adding random noise to a hinge function shown in the image below:

I then fitted a catboost regressor, with iterations=1 and depth=1. My understanding is this should split the x values into two leaf nodes, and the prediction is the mean of the y values in each node. My expectation is the model will look like the image below - this has a mean squared error of ~225:

However catboost split at ~30 and the predicted value in each split doesn't appear to be the mean of the blue points in each split - the mean squared error is ~1240:

My code is:

# generate data
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

x = np.linspace(0,40,1000)
e = np.random.normal(0,5,1000)
y = 10 + np.where(x>18,3*(x-18),0) + e

plt.figure();
plt.scatter(x,y, s=1);
plt.plot(x, 10 + np.where(x>18,3*(x-18),0), color = 'orange', alpha=0.7);

# expected model
y_pred = np.where(x>18,y[x>18].mean(),y[x<=18].mean())

plt.figure();
plt.scatter(x, y, s=1);
plt.plot(x, y_pred, color='orange');

print(f'mse: {mean_squared_error(y, y_pred):.2f}')

# catboost model
from catboost import CatBoostRegressor, Pool
from sklearn.metrics import mean_squared_error

train_pool = Pool(x, y)
estimator = CatBoostRegressor(n_estimators=1, max_depth=1, loss_function='RMSE')
estimator.fit(train_pool)

y_pred = estimator.predict(x)

plt.figure();
plt.scatter(x, y, s=1);
plt.plot(x, y_pred, color='orange');

print(f'mse: {mean_squared_error(y, y_pred):.2f}')

Answer 1

I think you have the right idea but you are somewhat thrown off by default values of the learning rate.

All gradient boosting algorithms make the working assumption that they will be used iteratively so they define a learning rate/step size. This is fundamental to all boosting algorithms and it almost always a tunable parameter. Friedman et al. (2001) seminal boosting paper "Additive logistic regression: a statistical view of boosting" refer to it as a shrinkage factor $\nu$ and it effectively regularises the learning by not allowing the learners to over-fit the data (Rossett et al. 2004 "Boosting as a regularized path to a maximum margin classifier" explores this in some detail). Catboost, as well as XGBoost, refer to the learning rate as $\eta$. As a general rule, learning rates are purposely set to low values such that the boosting procedure is able to learn the final function in a principled incremental way. The hard truth is that the smaller value $\eta$ help us to overcome overfitting at the expense of requiring more iterations during training.

Having said the above, let's go back to Catboost in particular. If during the call to Catboost we do not specify the learning rate, Catboost tries to pick it automatically based on some internal heuristics that are a function of the dataset and the number of iterations. For this use, which in all fairness is an edge-case having only a single iteration, the learning rate is automatically choose to be 0.02999999933 (Catboost version 0.9.1.1). Based on your attached graph this learning rate in not aggressive enough. If we make a call like:

estimator = CatBoostRegressor(n_estimators=1, max_depth=1, loss_function='RMSE',  
                              learning_rate=1 )

the plot of the "final fit", will indeed match your expectations and will look like:

This model has a MSE of ~105.5; Catboost took a single but quite well-educated step! :)