# Computation of log-loss optimal function

I am new to machine learning. Though I know the basic concept of loss functions, Bayes optimal function, Bayes Risk etc. as well as the fundamental concepts of probability, calculus and so on, I can’t seem to figure out the following exercise:

Let $Y_I = \{-1,1\}$ and $f: X \to \mathbb{R}$. Compute explicitly the minimizer $f_L(x)$ of the logistic loss $$L(y, f(x)) = \text{log}(1 + \text{exp}(-y\,f(x)))$$ in dependency of $\mathbb{P}(Y = 1| X = x)$.

What do I have to do here? Obviously I have to minimize some function but minimizing the loss function itself does not work. I revisited the lecture notes and apparently it is $$\arg\max\limits_{y} \mathbb{E}[L(y, f(x)), \mathbb{P}(Y = 1|X = x)].$$ This is equivalent to $$\arg\max\limits_y \int_{\mathbb{R}} L(y,f(x))\mathbb{P}(Y=1|X=x),$$ right? But what is $\mathbb{P}(Y=1|X=x)$ exactly? And how to proceed?

To my surprise, I completely forgot about this question. In the meantime I came across a solution that I will sketch below.

The objective is to minimize $$\mathbb{E}[L(Yf(x))\, \vert\, X = x]$$.

Now rewrite $P = P(Y = 1\, \vert\, X = x)$ and $f = f(x)$. Then we calculate that

\begin{align} \frac{\partial}{\partial f} \mathbb{E}_Y[L(Yf)\, \vert\, X = x] &= \frac{\partial}{\partial f} \Big(PL(f) + (1-P)L(-f)\Big) \\ &= PL'(f) - (1-P)L'(f) \\ &= P\frac{-e^{-f}}{1 + e^{-f}} + (1-P)\frac{e^{f}}{1 + e^{f}} \\ &= P\frac{-e^{-f}}{1 + e^{-f}} + (1-P)\frac{1}{1 + e^{-f}} \\ &\overset{!}{=} 0 \end{align}

and thus after solving for $f$ we see that $$f = \log\frac{P}{1 - P}$$ As a matter of fact we are done now since $L(f)$ is convex and hence also some composition. As such the critical point is definitely a minimum. I hope that helps!

• What is the interpretation of this result - and what is $\alpha$?
– Pugl
Aug 2, 2017 at 21:45
• Sorry I mixed up some notation. At first we used $\alpha$ to derive the solution, I changed it to $f$ for the sake of clarity. Aug 3, 2017 at 6:27