2
$\begingroup$

If I understood correctly the XGBoost is a framework that operates on gradient tree boosting. It means that behind the scenes, it uses a decision tree to make a prediction. So, from what I read in the Introduction to Statistical Learning by G.James et al., I should obtain a finite number of predicted values. This deduction comes from a particular fragment (chapter 8, p. 306), namely:

  1. We divide the predictor space—that is, the set of possible values for X1,...,Xp into J distinct and non-overlapping regions, R1,...,RJ.
  2. For every observation that falls into the region Rj, we make the same prediction, which is simply the mean of the response values for the training observations in Rj.

Suppose that in Step 1, we obtain two regions and that the response mean of the training observations in the first region is 10, while the response mean in the second region is 20. Then for a given observation X = x, if x ∈ R1, we will predict a value of 10, and if x ∈ R2, we will predict a value of 20.

After having trained the model with maximum depth of 10 (that would give 1024 distinct nodes), I made predictions on a test set and I found out that I have 524000 distinct values which means that I have a unique value for each row in the dataset.

In the light of the quoted fragment, my question is - why is it so? Why are all my predicted values distinct?

Here's my python code, if it's needed:

XGB_PARAMETERS = {
    'eta': 0.2, 
    'nthread': 4, 
    'min_child_weight': 30, 
    'max_depth': 10,
    'colsample_bytree': 0.7, 
    'subsample': 0.8,        
    'eval_metric': 'rmse',
    'silent':1
 }

model = xgb.train(
    XGB_PARAMETERS,
    d_test,
    150,
    watchlist,
    early_stopping_rounds=10,
    maximize=False,
    verbose_eval=10)

d_final_test = xgb.DMatrix(test[f2].values)
ypred = model.predict(d_test)
y_test = model.predict(d_final_test)
test['duration'] = np.exp(y_test) - 1
$\endgroup$
  • $\begingroup$ You mean, how I checked that all values are distinct? I just executed print(len(test['duration'].unique())) $\endgroup$ – mångata Feb 10 at 19:51
  • 2
    $\begingroup$ Your trees may have a maximum depth of 10, but you have 150 of them, and the final prediction is a weighted sum of each of the 150 individual trees' predictions... that gives a lot of possible values as the output, even allowing for correlation between trees. $\endgroup$ – jbowman Feb 10 at 21:35
2
$\begingroup$

This code is defining a GBM of 150 trees with maximum depth 10. The predicted values are therefore not based on a single (rather deep) tree but actually on the ensemble of 150 trees. Given that each tree can potentially estimate $1024$ different values theoretically we can have up to $1024^{150}$ distinct values which is a relatively affluent number. I would suggest using a very small number of leaves as well as trees and then slowly increase the number of trees; that way the recursive partitioning of the sample space (through piece-wise flat functions) is will be easier to visualise.

$\endgroup$
0
$\begingroup$

Perhaps it's because you are using subsample, which means that you are enabling stochastic gradient descent. See the parameters documentation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.