So, I guess I am completely lost on subject of regression trees. Here is my code:

import pandas as pd
import xgboost as xgb
import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0, 10, 20)
train_X = pd.DataFrame({"A":x})
train_y = pd.Series(np.sin(x))

gbm = xgb.XGBClassifier(max_depth=7, n_estimators=300, 
learning_rate=0.01).fit(train_X, train_y)
predictions = gbm.predict(train_X)
plt.plot(x, predictions)

The above code generates straight line as prediction. I expected some step function approximating sin. Any suggestions? How do regression trees work?


1 Answer 1


With a max_depth=7 and n_estimators=300 you have given the model $300 \times 2^7 = 38400$ opportunities to split on your predictor. This is well beyond the number of pixels of vertical resolution available to show a stair step shape in any graph (my macbook pro is 2880-by-1800 for the entire monitor).

Crank down those values and you'll see the stairstep.

  • $\begingroup$ gbm = xgb.XGBClassifier(max_depth=2, n_estimators=10, learning_rate=0.01).fit(train_X, train_y) produces the same result. What parameter would you recommend? $\endgroup$ Aug 21, 2016 at 22:14
  • $\begingroup$ That is very surprising, I would think that, given all your code is correct, you would certainly see the starstep nature with those parameters. Are you sure you should be using a classifier with regression type data? $\endgroup$ Aug 21, 2016 at 22:16
  • $\begingroup$ Of course, I am not sure. The goal is to understand how regression trees work, rather than get good approximation. I am even more surprised with the following example. Replace train_y = pd.Series(np.sin(x)) with train_y = pd.Series(x+5). It generates strait line at 5, while I would imagine least squares should produce straight line at 10. Can you try to reproduce the above example on your computer? I just want to make sure that I do not have any bugs in code. $\endgroup$ Aug 21, 2016 at 22:21
  • $\begingroup$ I can try to reproduce a little later in the day. In the meantime, I think XGBRegressor is more appropriate for your example. $\endgroup$ Aug 21, 2016 at 22:22
  • 1
    $\begingroup$ I doubt there is a bug in xgboost, that's a library used by thousands of people and you would know about it if one of its fundamental features was broken. $\endgroup$ Aug 21, 2016 at 22:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.