4
$\begingroup$

I am trying to learn from Hastie on boosting methods. A question has bothered me for weeks. The book describes Forward-Stagewise Fitting, that is, fit a weak learner to explain the residuals from the existing learner. In Forward-Stagewise Fitting, you can either:

(i) Use a shallow tree to fit the residuals from the last iteration, or

(ii) Compute the negative gradient of the loss function and then fit a shallow tree to the gradient.

The second choice (ii) here is called Gradient Boosting.

The book describes the first choice (i) as becoming impractical for a general loss function (e.g., not squared-error loss), and thus we need Gradient Boosting. I cannot understand this part. It seems to me that fitting a tree directly to the residuals is entirely possible even with loss functions like absolute error or Huber, and it will be no different from the case with the squared-error loss function. Am I missing something?

Thanks in advance.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

First, let's make clear here that the base learners most often don't know about the loss function: they most often do not try to minimize it. The loss function is from the boosting part, the base learners are agnostic to it. The only way they can be exposed to the loss function is through the surrogate targets we train them on. The problem with Bernoulli I will expose below is stereotypical, but under any non-Gaussian log likelihood you'll get a divergence. If you simply use the residuals, you are not actually using the loss function, unless it was actually Gaussian.

Consider the Bernoulli loss function:

$$\mathcal L(\hat y_1,y) = -\log(\hat y_1)y-\log(1-\hat y_1)(1-y)$$

Having obtained $\hat y_1$, the residuals are $\epsilon_1=y-\hat y_1$. The residuals simply cannot be Bernoulli distributed, primarily because they are continuous, as $\hat y_1$ is often a proportion (assuming a decision tree base learner), but also that the sum $\hat y_1+\hat y_2$ can exceed $0$ or $1$, since residuals can be negative. As weak learners display a lot of bias, they could predict first small $\hat y_1$, but then negative $\hat y_2$, making $\hat y_1+\hat y_2<0$. The converse is also true, where they could make $\hat y_1 \approx 1$, but then $\hat y_2>0$ and you may reach a point where $\hat y_1+\hat y_2>1$.

If the loss function was Normal, then we could keep on fitting to residuals, because they likely would continue to follow a Normal distribution.

The motivation to use the pseudo-residuals based on the gradient is that, for the Normal negative loglikelihood case (which is simply the mean squared error), the simple residuals are precisely the negative gradient of the loss function.

It can be shown that, when you predict the pseudo-residuals, you "travel" in the direction that minimizes the loss function. If you use direct residuals, however, since they are the negative gradient of the Gaussian loglikelihood, you will optimize the Gaussian likelihood, not your loss function.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks. I can see that for the Bernoulli loss function, method (i) will be quite difficult. But for absolute loss or Huber loss, it seems to me that method (i) will continue to work? Hastie somehow writes that using method (i) under absolute or Huber loss will be difficult too. Is it because training a tree under squared loss is generally much faster than under other losses (e.g., absolute, Bernoulli, Huber), maybe because finding the split under squared loss involves less computation? $\endgroup$
    – max sim
    Commented Apr 7, 2021 at 22:26
  • 1
    $\begingroup$ @maxsim First, let's make clear here that the base learners most often don't know about the loss function: they most often do not try to minimize it. The loss function is from the boosting part, the base learners are agnostic to it. In general, computing absolute loss or Huber loss for that matter is not sensibly different from computing the squared error. The problem with Bernoulli is stereotypical, but under any non-Gaussian log likelihood you'll get a divergence. I would like to make clear here that, if you simply use the residuals, you are not actually using the loss function. $\endgroup$
    – Firebug
    Commented Apr 7, 2021 at 22:30
  • $\begingroup$ To see myself really understand what you are saying, the forward-stagewise algorithm minimizes the sum of L(y_i, G(x_i) + g(x_i)) across i. Here, L is the loss function and G is the accumulated learners so far, and g is the current learner to be trained. This is different from minimizing L(y_i - G(x_i), g(x_i)) under Bernoulli loss. (I realize I was not being accurate saying "use a shallow tree to fit the residuals from the last iteration" in my question.) $\endgroup$
    – max sim
    Commented Apr 8, 2021 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.