R Package GBM - Bernoulli Deviance

Question

All,

I am trying to study the GBM package in R.

I. I wanted to try and figure out where the deviance, initial value, gradient and terminal node estimates came from. Please see this snippet:

To start out, I was not sure where the Bernoulli deviance came from. I expected that the deviance, would be -2 multiplied by the log-likelihood

, ignoring the weight variable w. Can anyone suggest what I am doing (or missing) wrong to dervive what is shown in GBM?

II. I was also confused on where the terminal node estimates came from?

Any help is very appreciated!

Brian

Answer 1

It is a mathematical trick. We have

\begin{align*} \log\frac{p_i}{1-p_i}=f(x_i) \end{align*} and from this we get \begin{align*} \frac{1}{1-p_i}=1+\exp(f(x_i)) \end{align*}

The log likelihood is \begin{align*} \sum_{i=1}^n\left[y_i\log(p_i)+(1-y_i)\log(1-p_i)\right]&=\sum_{i=1}^n\left[y_i\log\frac{p_i}{1-p_i}+\log(1-p_i)\right]\\ &=\sum_{i=1}^n\left[y_if(x_i)-\log\frac{1}{1-p_i}\right]\\ &=\sum_{i=1}^n\left[y_if(x_i)-\log(1+\exp(f(x_i)))\right]\\ \end{align*}

Only some terms were rearranged. I hope I made clear how exactly it was done.

Answer 2

I'm also studying the GBM package!

1. mpiktas, i think you forgot a log on the left hand side in the 2nd equation? I assume you substitute p_i with 1/(1+exp(-f(x_i))), but then in the 2nd row above there is log(1/...) = log(1+...), or am i wrong? Anyway, i think then you did it right in the third row...

2. Can you tell us what is the motivation for choosing p_i = 1/(1+exp(-f(x_i)))? Where does that come form? The p_i should mirror the proportion of success, i.e. class proportions, right?

Thanks! peter