# GBM and 1-dim approximation

So, I guess I am completely lost on subject of regression trees. Here is my code:

import pandas as pd
import xgboost as xgb
import matplotlib.pyplot as plt
import numpy as np

x = np.linspace(0, 10, 20)
train_X = pd.DataFrame({"A":x})
train_y = pd.Series(np.sin(x))

gbm = xgb.XGBClassifier(max_depth=7, n_estimators=300,
learning_rate=0.01).fit(train_X, train_y)
predictions = gbm.predict(train_X)
plt.plot(x, predictions)
plt.show()


The above code generates straight line as prediction. I expected some step function approximating sin. Any suggestions? How do regression trees work?

With a max_depth=7 and n_estimators=300 you have given the model $300 \times 2^7 = 38400$ opportunities to split on your predictor. This is well beyond the number of pixels of vertical resolution available to show a stair step shape in any graph (my macbook pro is 2880-by-1800 for the entire monitor).
• gbm = xgb.XGBClassifier(max_depth=2, n_estimators=10, learning_rate=0.01).fit(train_X, train_y) produces the same result. What parameter would you recommend? Aug 21, 2016 at 22:14
• Of course, I am not sure. The goal is to understand how regression trees work, rather than get good approximation. I am even more surprised with the following example. Replace train_y = pd.Series(np.sin(x)) with train_y = pd.Series(x+5). It generates strait line at 5, while I would imagine least squares should produce straight line at 10. Can you try to reproduce the above example on your computer? I just want to make sure that I do not have any bugs in code. Aug 21, 2016 at 22:21
• I can try to reproduce a little later in the day. In the meantime, I think XGBRegressor is more appropriate for your example. Aug 21, 2016 at 22:22