How do you prove that minimizing the binomial deviance estimates the log-odds? i.e:

$$ \ln{\left ( \frac{p(x_i)}{1-p(x_i)} \right )} = \underset{f(x_i)}{argmin} \ \mathbb{E} \left [y_i \ln \left ( \frac{e^{f(x_i)}}{1+e^{f(x_i)}} \right ) + (1-y_i) \ln \left ( 1-\frac{e^{f(x_i)}}{1+e^{f(x_i)}} \right ) \right ] $$


$$ y_i \in \{ 0,1 \} \\ p(x_i)=\frac{e^{f(x_i)}}{1+e^{f(x_i)}} $$


According to Hasti, Tibshirani, Friedman; Elements of Statistical Learning 2nd Ed, the minimizer should be $1/2$ of the log odds, just like for exponential loss, however no proof is provided.

A proof for the minimizer of the exponential loss (not the binomial deviance) is derived here by Weatherwax and Epstein, which is straight-forward and is based on the $\{-1,1\}$ notation. However the same principles don't seem to apply (?) to the binomial deviance with the $\{0,1\}$ notation, since one term becomes 0 in the expectancy.

A statement is made at this wikipedia page and some proof instruction is somewhat given, however I find the notation confusing. The coding there is $\{-1,1\}$ too (I think).

Any proof/ intuition/ idea of how to go about this would be much appreciated.


1 Answer 1


So it seems the solution is much simpler than I though. Hasti, Bishirani and Friedman in the source mentioned above mention the gradient of the (multinomial) deviance, which is:

$$ g_{ik} = -I(y_i=G_k)+p_k(x_i) $$

This holds for binomial deviance as well, and is fairly easy to show that:

$$ \frac{\partial L(y_i,f(x_i))}{\partial f(x_i)} = \frac{\partial}{\partial f(x_i)} \left ( y_i \ln \left ( \frac{e^{f(x_i)}}{1+e^{f(x_i)}} \right ) + (1-y_i) \ln \left ( 1-\frac{e^{f(x_i)}}{1+e^{f(x_i)}} \right ) \right ) =-y_i+p(x_i) $$

By Leibniz' integration rule it holds that $\frac{\partial}{\partial f(x_i)} \mathbb{E}[\cdot]=\mathbb{E}[\frac{\partial}{\partial f(x_i)}( \cdot)]$, thus: $$ \frac{\partial \mathbb{E}[L(y_i,f(x_i))]}{\partial f(x_i)} = \mathbb{E} \left [ \frac{\partial L(y_i,f(x_i))}{\partial f(x_i)} \right ] \overset{!}{=}0 \\ \Leftrightarrow \mathbb{E}[-y_i+p(x_i)] \overset{!}{=} 0 \\ \Leftrightarrow (-y_i+p(x_i)) P(y_i=1)+(-y_i+p(x_i)) P(y_i=0) \overset{!}{=} 0 \\ \Leftrightarrow (-1+p(x_i)) P(y_i=1)+(-0+p(x_i)) P(y_i=0) \overset{!}{=} 0 \\ \Leftrightarrow (-1+p(x_i)) P(y_i=1)+(-0+p(x_i)) P(y_i=0) \overset{!}{=} 0 \\ \Leftrightarrow p(x_i)=\frac{P(y_i=1)}{P(y_i=0)} \\ \Leftrightarrow \underbrace{\frac{e^{f(x_i)}}{1+e^{f(x_i)}}}_{=(1+e^{f(x_i)})^{-1}} =\frac{P(y_i=1)}{P(y_i=0)} \\ \Leftrightarrow f(x_i)=\ln \left ( \underbrace{\frac{P(y_i=1)}{P(y_i=0)}}_{=log-odds} \right ) $$


This site is temporarily in read-only mode and not accepting new answers.

Not the answer you're looking for? Browse other questions tagged .