The logistic loss is given by: $l(y, z) = \log(1+\exp(-y\,z))$. Show that the optimal classifier for the logistic loss is given by: $\phi(x) = \log(\eta(x)) - \log(1-\eta(x))$. More formally, for a random variable $Y \in \{-1, 1\}$, show that the following expression is minimized by $\phi$:

$$\min_{f}\mathbb E [l(Y, f(X)]$$

Where the minimization is done over all measurable functions from the domain of $X$ to $\mathbb R$.

What I've tried

It's straightforward to break this down by considering the conditional expectation and applying the tower property of expectation, but the next step (the minimization over all measurable functions) is where I'm having trouble. It seems to require some calculus of variations.


1 Answer 1


This doesnt need calculus of variations. Write the loss, pointwise conditional on $X$:

$$\eta(x)l(c) + (1-\eta(x))l(-c)$$

Now minimize this expression with respect to the pointwise prediction $c$. This gives the desired expression for the bayes classifier.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.