7
$\begingroup$

All,

I am trying to study the GBM package in R.

I. I wanted to try and figure out where the deviance, initial value, gradient and terminal node estimates came from. Please see this snippet:

alt text

To start out, I was not sure where the Bernoulli deviance came from. I expected that the deviance, would be -2 multiplied by the log-likelihood

alt text

, ignoring the weight variable w. Can anyone suggest what I am doing (or missing) wrong to dervive what is shown in GBM?

II. I was also confused on where the terminal node estimates came from?

Any help is very appreciated!

Brian

$\endgroup$
1
  • $\begingroup$ You can write in LaTeX on this site, just put the code in $ $. $\endgroup$
    – user88
    Dec 14, 2010 at 9:38

2 Answers 2

10
$\begingroup$

It is a mathematical trick. We have

\begin{align*} \log\frac{p_i}{1-p_i}=f(x_i) \end{align*} and from this we get \begin{align*} \frac{1}{1-p_i}=1+\exp(f(x_i)) \end{align*}

The log likelihood is \begin{align*} \sum_{i=1}^n\left[y_i\log(p_i)+(1-y_i)\log(1-p_i)\right]&=\sum_{i=1}^n\left[y_i\log\frac{p_i}{1-p_i}+\log(1-p_i)\right]\\ &=\sum_{i=1}^n\left[y_if(x_i)-\log\frac{1}{1-p_i}\right]\\ &=\sum_{i=1}^n\left[y_if(x_i)-\log(1+\exp(f(x_i)))\right]\\ \end{align*}

Only some terms were rearranged. I hope I made clear how exactly it was done.

$\endgroup$
2
  • $\begingroup$ Thanks! I did not get the connection that f(x) was the logit transform. Did you happen to have insight into the terminal node -- should that be self explanatory to me? It looks like GBM uses a regression tree for classification (versus the classification algorithm shown in Elements of Statistical Learning. Is that your understanding as well? Thanks again! $\endgroup$
    – B_Miner
    Dec 14, 2010 at 23:14
  • $\begingroup$ I am not very familiar with machine learning. For what I gathered from the vignette of GBM, yes GBM uses regression tree for classification, but regression trees are just one of classification algorithms. The terminal node formula is used to calculated the addition to regression function, during the learning process. It is used everytime after classification. I suggest to formulate separate question to get a better answer. $\endgroup$
    – mpiktas
    Dec 15, 2010 at 7:16
0
$\begingroup$

I'm also studying the GBM package!

  1. mpiktas, i think you forgot a log on the left hand side in the 2nd equation? I assume you substitute p_i with 1/(1+exp(-f(x_i))), but then in the 2nd row above there is log(1/...) = log(1+...), or am i wrong? Anyway, i think then you did it right in the third row...

  2. Can you tell us what is the motivation for choosing p_i = 1/(1+exp(-f(x_i)))? Where does that come form? The p_i should mirror the proportion of success, i.e. class proportions, right?

Thanks! peter

$\endgroup$
1
  • $\begingroup$ Yes, I forgot the log on the left hand side, or rather I added unnecessary log on the right hand side. Motivation for $p_i =1/(1+\exp(f(x_i)))$s comes from the fact, that $f(x_i)$ is unbounded, but $p_i$ is constrained to interval $(0,1)$, so we need to transform $f(x_i)$ in order to succesfully model $p_i$. This transformation does exactly that. Look for more explanation in wikipedia under logit function. $\endgroup$
    – mpiktas
    Dec 14, 2010 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.