I'm trying to understand how regression trees work, I've been experimenting with catboost and xgboost in python, and I'm getting results which I don't expect, can someone please clarify (and apologies in advance if this is a coding error)

I've generated test data by adding random noise to a hinge function shown in the image below:

I then fitted a catboost regressor, with iterations=1 and depth=1. My understanding is this should split the x values into two leaf nodes, and the prediction is the mean of the y values in each node. My expectation is the model will look like the image below - this has a mean squared error of ~225:

However catboost split at ~30 and the predicted value in each split doesn't appear to be the mean of the blue points in each split - the mean squared error is ~1240:

My code is:

# generate data
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

x = np.linspace(0,40,1000)
e = np.random.normal(0,5,1000)
y = 10 + np.where(x>18,3*(x-18),0) + e

plt.scatter(x,y, s=1);
plt.plot(x, 10 + np.where(x>18,3*(x-18),0), color = 'orange', alpha=0.7);

# expected model
y_pred = np.where(x>18,y[x>18].mean(),y[x<=18].mean())

plt.scatter(x, y, s=1);
plt.plot(x, y_pred, color='orange');

print(f'mse: {mean_squared_error(y, y_pred):.2f}')

# catboost model
from catboost import CatBoostRegressor, Pool
from sklearn.metrics import mean_squared_error

train_pool = Pool(x, y)
estimator = CatBoostRegressor(n_estimators=1, max_depth=1, loss_function='RMSE')

y_pred = estimator.predict(x)

plt.scatter(x, y, s=1);
plt.plot(x, y_pred, color='orange');

print(f'mse: {mean_squared_error(y, y_pred):.2f}')
  • 1
    $\begingroup$ Keep the big picture in mind. The choice of basis functions is all-important. Recursive partitioning is in effect using basis functions for continuous predictors that are piecewise flat with unknown jump points. The next less restrictive basis is a linear spline, i.e., piecewise straight lines with no gaps and with unknown knots, i.e., points of slope changes. Cubic splines are even better. Recursive partitioning IMHO does not have much to offer for continuous Xs, and resulting fits are not realistic. $\endgroup$ – Frank Harrell Dec 8 '18 at 14:19
  • $\begingroup$ Thanks @FrankHarrell, I understand - originally I had assumed regression trees partitioned data and then applied some form of regression to the data in the leaf nodes i.e.a linear fit which I now know was a misconception. What I'm really looking for is something which can recursively partition data with a large number of categorical data (specifically time of day, week of year etc) then fit some sort of piece wise linear or spline to the partitioned data - do you have any recommendations? I've done this manually (via a heuristic) which works well $\endgroup$ – David Waterworth Dec 9 '18 at 2:57
  • $\begingroup$ What you need seems to be a well formulated regression model with flexible modeling of smooth nonlinear effects, using for example restricted cubic splines. The data you called categorical are really continuous and should be modeled aday of week is mixed continuous/categorical). More details are in my RMS course notes. $\endgroup$ – Frank Harrell Dec 9 '18 at 4:26
  • $\begingroup$ Thanks for your help @FrankHarrell, I'll tale a look (I've used your rms package before). The reason I consider the temporal variables categorical is I'm modelling building heating / cooling / dehumidifying electrical load. It's can be modelled well be regression splines but the response changes in jumps at certain times of the day (i.e. 9am - 5pm has high sensitivity, evening lower, night not much response etc.). What I'm trying to do is automatically learn this regime for different sites. (i.e. one may be 7am - 4pm mon-Fri, another 9am-5pm 7 days a week etc.) $\endgroup$ – David Waterworth Dec 9 '18 at 5:16
  • $\begingroup$ Good point. If you think the jump may only be in the slope and not in the mean, then a linear spline may do the trick. $\endgroup$ – Frank Harrell Dec 9 '18 at 12:57

I think you have the right idea but you are somewhat thrown off by default values of the learning rate.

All gradient boosting algorithms make the working assumption that they will be used iteratively so they define a learning rate/step size. This is fundamental to all boosting algorithms and it almost always a tunable parameter. Friedman et al. (2001) seminal boosting paper "Additive logistic regression: a statistical view of boosting" refer to it as a shrinkage factor $\nu$ and it effectively regularises the learning by not allowing the learners to over-fit the data (Rossett et al. 2004 "Boosting as a regularized path to a maximum margin classifier" explores this in some detail). Catboost, as well as XGBoost, refer to the learning rate as $\eta$. As a general rule, learning rates are purposely set to low values such that the boosting procedure is able to learn the final function in a principled incremental way. The hard truth is that the smaller value $\eta$ help us to overcome overfitting at the expense of requiring more iterations during training.

Having said the above, let's go back to Catboost in particular. If during the call to Catboost we do not specify the learning rate, Catboost tries to pick it automatically based on some internal heuristics that are a function of the dataset and the number of iterations. For this use, which in all fairness is an edge-case having only a single iteration, the learning rate is automatically choose to be 0.02999999933 (Catboost version Based on your attached graph this learning rate in not aggressive enough. If we make a call like:

estimator = CatBoostRegressor(n_estimators=1, max_depth=1, loss_function='RMSE',  
                              learning_rate=1 )

the plot of the "final fit", will indeed match your expectations and will look like: enter image description here

This model has a MSE of ~105.5; Catboost took a single but quite well-educated step! :)

  • $\begingroup$ Thanks, I hadn't really paid much attention to the learning rate! $\endgroup$ – David Waterworth Dec 9 '18 at 2:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.