When we are modeling non-negative data with linear regression we often log-transform the dependent, thereby ensuring that all the predictions made by the model will be positive as well.

Does it make sense to do the same when using Gradient Boosted Trees (GBTs)? For Random Forests the output is an average of the training labels in the "winning" leaf of each tree, thereby ensuring that the prediction will be non-negative.

Is it safe to make the same assumption for GBTs? I haven't been able to follow the derivation from the original paper (section 4.3).

  • $\begingroup$ It surprises me that you can get negative values from GBTs; my understanding of this was that you couldn't predict outside the range of the observed (all positive) data. $\endgroup$ – zbicyclist Feb 21 '17 at 15:38
  • $\begingroup$ You can definitely get negative predictions from a GBT even if all the training data is positive. $\endgroup$ – Matthew Drury Feb 21 '17 at 15:54
  • $\begingroup$ @MatthewDrury is there some way to show this mathematically? Even a toy example would work. $\endgroup$ – Bar Feb 22 '17 at 10:54
  • $\begingroup$ A single tree, or ensemble of trees, trained only by splitting the observed data alone would not be able to predict outside the range of the data (e.g. CART or random forests). However, boosting creates trees fitted to the residuals of previous model predictions. $\endgroup$ – Paul Feb 22 '17 at 19:34
  • $\begingroup$ @Bar I got a chance to write an answer for you. I hope this helps. $\endgroup$ – Matthew Drury Feb 24 '17 at 4:38

As I mentioned in the comments:

You can definitely get negative predictions from a GBT even if all the training data is positive.

Here's a simple example of this in practice:

x = np.array([0, 1, 2, 3]).reshape(-1, 1)
y = np.array([0, 100, 0, 100]).reshape(-1, 1)

gbr = GradientBoostingRegressor(learning_rate=1.0, n_estimators=2, max_depth=1)
gbr.fit(x, y)

Of course, the parameters here are extreme so as to make the point in the simplest possible situation (no one should use a learning rate of 1.0 in practice). The predictions from this model produce negative values at the second stage:

for i, preds in enumerate(gbr.staged_predict(x)):
    print("Preds at stage {}: {}".format(i, preds))

which produces the following output:

Preds at stage 0: [  0.          66.66666667  66.66666667  66.66666667]
Preds at stage 1: [ -11.11111111   55.55555556   55.55555556  100.       ]

To understand what happened, we need the gradient from the first stage:

for i, preds in enumerate(gbr.staged_predict(x)):
    grad = y.reshape(-1) - preds
    print("Gradient at stage {}: {}".format(i, grad))

Which produces:

Gradient at stage 0: [  0.          33.33333333 -66.66666667  33.33333333]
Gradient at stage 1: [ 11.11111111  44.44444444 -55.55555556   0.        ]

Recall the a regression tree is fit to the gradient at stage zero, and the average residual is calculated in each of their terminal regions of this true to form the update step of the booster. This tree resulted in two terminal regions, the first consisting of the first three observations, and the second of only the last observation. The average gradient (residual) in this first region is 11.111..., which is then subtracted from the prediction of 0.0 for the first observation, resulting in -11.111....

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.