componentwise boosting dates back at least to Bühlman and Yu (2003), where in each boosting iteration a set of base-learners (e.g. simple linear models) depending on a subset of the covariates are used. This implies some variable selection.

Let us consider the case, in which we use base-learners with only one covariate. If we have $P$ covariates, we then have $P$ base learners in each boosting iteration. At the end of each boosting iteration, we select the best base-learner among this $P$ base-learners for the update. Since only one covariate is in cluded in each base-learner, we update only the parameters for one covariate.

In the literature one can find componentwise boosting based on functional gradiend descent on the one hand and componentwise boosting based on one-step Fisher scoring (e.g. Tutz and Binder 2006; Tutz and Binder 2007 ...) on the other hand. My question refers to the latter, in which a penalized maximum likelihood estimation (ridge penalty) is used to update the parameters by one-step Fisher scoring.

Here comes my question: If we have only metric covariates and use componentwise boosting with only one single predictor in each step, do we then need penalized maximum likelihood estimation?

My guess is no, because in my view penalization is only needed if we use componentwise boosting, where we want to update a set of predictors in each step (and not only one single predictor). Another reason where I see the need for penalization is the case, where we have a single multicategorial covariate (with $K$ categories) that has to be updated. Each multicategorial covariate has $K-1$ associated parameters and thus updating this multicategorial covariate means updating all $K-1$ associated parameters. Here, one can do penalization to update only a subset of the $K-1$ associated parameters, namely the 'important' ones, while the 'unimportant' ones are shrinked towards zero due to penalization.

Since we want to update one single metric predictor in each boosting step, we don't need penalization, right?


1 Answer 1


I think the simple answer is that you don't need a penalized maximum likelihood estimation to use component-wise boosting. But, if you don't use penalized MLE, then you might find that boosting ends up (after many iterations) to be similar to other model fitting techniques, such as gradient descent.

When you use a penalized MLE, you are trying to limit the number of covariates selected in your model, by introducing a penalty based on the number of non-zero coefficients. Boosting is an excellent technique to quickly maximize that penalized MLE. It works better than gradient descent, because gradient descent will try to change all the coefficients at each iteration of the fitting process. Every time all the coefficients are changed, a big penalty is introduced because now all the variables are non-zero. In boosting, the penalty is gradually increased because the coefficients are changed to be non-zero one at a time, making it easier to compare the extra loss from the penalty with the added fit from the new non-zero coefficient.

The point is, you can use a non-penalized likelihood with boosting, but then you might as well use gradient descent. There are definitely special cases where the fit will be very different using the two fitting techniques, especially when you have a large number of covariates. But if you have a large number of covariates, then you will likely need a automated variable selection method, such as the penalized MLE.

I think I should clarify that when I say "gradient descent", what I really mean is the opposite of component-wise gradient descent, you could say "global" gradient descent. Specifically, all of the covariates are simultaneously modified within each iteration.

So, you can use non-penalized MLE with component-wise gradient descent (boosting), but what I am saying is that you will probably get the same result with "global" gradient descent if you do it that way.

Additionally, upon rereading your comment, I think I understand some of the confusion. I think that you understand that the penalty on the MLE is based on the number of non-zero covariates, so it acts to automate variable selection. But what I think you misunderstand is that the penalty isn't applied to each "learner" (covariate) individually, but on the global model.

So, imagine you have a model with three variables, and you are using boosting with a penalized MLE. So, on the first iteration you boost one variable, and then go to the next iteration. Now, the fitter has a choice, it can either try and boost one of the other two variables, which would incur a penalty because an additional parameter would be added, or it can boost the first variable, which may not give as much of a better fit. If you didn't have the penalty, then it would just go for the best fit no matter which variable it was. In this case, you might as well have just modified all the variables at once, instead of doing it in three iterations.

  • $\begingroup$ Aren't you wrong saying that "gradient descent will try to change all the coefficients" (see chap. 4.1 in arxiv.org/pdf/0804.2752.pdf, where gradient descent is used to update only 'one variable' in each boosting step - because the gradient is based on only one variable)? My question referred to the case, where we want to update only one variable in each boosting step (e.g. using base-learners with only one variable). Then penalized MLE is not necessary because we have to estimate/update only one parameter at the end of a boosting step. Or do I have misunderstood something? $\endgroup$
    – Giuseppe
    Sep 9, 2013 at 13:02
  • $\begingroup$ @Giuseppe That paper is referring to the gradient descent within boosting, not the "typical" (more familiar to statisticians) gradient descent in which all of the covariates are simultaneously modified. I have tried to clarify in my answer. $\endgroup$
    – nograpes
    Sep 9, 2013 at 13:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.