3
$\begingroup$

I am new to machine learning. Though I know the basic concept of loss functions, Bayes optimal function, Bayes Risk etc. as well as the fundamental concepts of probability, calculus and so on, I can’t seem to figure out the following exercise:

Let $Y_I = \{-1,1\}$ and $f: X \to \mathbb{R}$. Compute explicitly the minimizer $f_L(x)$ of the logistic loss $$ L(y, f(x)) = \text{log}(1 + \text{exp}(-y\,f(x)))$$ in dependency of $\mathbb{P}(Y = 1| X = x)$.

What do I have to do here? Obviously I have to minimize some function but minimizing the loss function itself does not work. I revisited the lecture notes and apparently it is $$ \arg\max\limits_{y} \mathbb{E}[L(y, f(x)), \mathbb{P}(Y = 1|X = x)].$$ This is equivalent to $$\arg\max\limits_y \int_{\mathbb{R}} L(y,f(x))\mathbb{P}(Y=1|X=x),$$ right? But what is $\mathbb{P}(Y=1|X=x)$ exactly? And how to proceed?

$\endgroup$

1 Answer 1

1
+50
$\begingroup$

To my surprise, I completely forgot about this question. In the meantime I came across a solution that I will sketch below.

The objective is to minimize $$\mathbb{E}[L(Yf(x))\, \vert\, X = x]$$.

Now rewrite $P = P(Y = 1\, \vert\, X = x)$ and $f = f(x)$. Then we calculate that

$$ \begin{align} \frac{\partial}{\partial f} \mathbb{E}_Y[L(Yf)\, \vert\, X = x] &= \frac{\partial}{\partial f} \Big(PL(f) + (1-P)L(-f)\Big) \\ &= PL'(f) - (1-P)L'(f) \\ &= P\frac{-e^{-f}}{1 + e^{-f}} + (1-P)\frac{e^{f}}{1 + e^{f}} \\ &= P\frac{-e^{-f}}{1 + e^{-f}} + (1-P)\frac{1}{1 + e^{-f}} \\ &\overset{!}{=} 0 \end{align} $$

and thus after solving for $f$ we see that $$f = \log\frac{P}{1 - P}$$ As a matter of fact we are done now since $L(f)$ is convex and hence also some composition. As such the critical point is definitely a minimum. I hope that helps!

$\endgroup$
2
  • $\begingroup$ What is the interpretation of this result - and what is $\alpha$? $\endgroup$
    – Pugl
    Aug 2, 2017 at 21:45
  • 1
    $\begingroup$ Sorry I mixed up some notation. At first we used $\alpha$ to derive the solution, I changed it to $f$ for the sake of clarity. $\endgroup$
    – Taufi
    Aug 3, 2017 at 6:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.