Understand the cost function in the GBM implementation of R

Question

I was looking through the various cost function(deviances) in GBM's implementation of R and got confused by the cost function. I had always thought that the cost function for bernoulli would be log likelihood or something that approximates log likelihood. However, it seems different in the R source code of GBM

   if (dist$name != "pairwise")
   {
      switch(dist$name,
             gaussian = weighted.mean((y - f)^2,w) - baseline,
             bernoulli = -2*weighted.mean(y*f - log(1+exp(f)),w) - baseline,
             laplace = weighted.mean(abs(y-f),w) - baseline,
             adaboost = weighted.mean(exp(-(2*y-1)*f),w) - baseline,
             poisson = -2*weighted.mean(y*f-exp(f),w) - baseline,
             stop(paste("Distribution",dist$name,"is not yet supported for method=permutation.test.gbm")))
   }

The above is the code from the GBM source code. Can someone explain the cost function in case of Bernoulli, and Poisson?

Answer 1

The entry under bernoulli is, in fact, proportional to the log-likelihood of the Bernoulli distribution. See my answer here: Scikit Binomial Deviance Loss Function

The poisson entry is the same. Writing $p$ for the pdf and $l$ for the log likelihood:

$$ p(x; \mu) = \frac{\mu^x}{x!} e^{-\mu} \Rightarrow l(x; \mu) = \sum_i x_i \log(\mu_i) - \mu_i - \log(x_i!) $$

The $\log(x_i!)$ is just a constant and can be ignored for the purposes of optimization. In gradient boosting, we fit our trees $f$ to the linear scale, and in Poisson regression the link function is the exponential, so we have

$$ \mu_i = e^{f_i} $$

So the loss function reduces to

$$ l(x; \mu) = \sum_i x_i f_i - e^{f_i} $$

Which is what is in the code.

Can you please explain what is f in case of poisson distribution?

Sure.

The symbol $f$ stands for the same thing in all cases: it is the current predictions from the model on the linear scale. This idea comes from the theory of Generalized Linear Models, but if you don't know that, it's still understandable. In the case of gradient boosting, $f$ stands in for the predictions from your sequence of regression trees:

$$ f(x_1, x_2, \ldots, x_k) = \sum_i T_i(x_1, x_2, \ldots, x_k) $$