# Is the log loss function $f(w) = y_t \log(y_p) + (1 - y_t) \log(1 - y_p)$ convex in $w$? [duplicate]

Kaggle defines the log loss function as: https://www.kaggle.com/wiki/LogarithmicLoss

$$f(w) = \log \Pr(y_t|y_p) = y_t \log(y_p) + (1 - y_t) \log(1 - y_p)$$

Let $y_t \in {0, 1}$, and $y_p$ is given by the sigmoid function $y_p = \sigma(w \cdot x) \in (0,1)$, where $\sigma$ is defined by

$$\sigma(w \cdot x) = \dfrac{1}{(1+\exp(-w \cdot x)}$$

$w,x \in \mathbb{R}^n$. The objective variable is $w$ so $x$ could be thought of as a weight.

So putting everything together:

$$f(w) = y_t \log\left(\dfrac{1}{(1+\exp(-w \cdot x)}\right) + (1 - y_t) \log\left(1 - \dfrac{1}{(1+\exp(-w \cdot x)}\right)$$

My question is: Is $f(w)$ convex?

I have read up some references but can't see the relationship: http://qwone.com/~jason/writing/convexLR.pdf. If anyone has a reference, it will be greatly appreciated!!

• And what is $y_t$? Could you rewrite $f(w)$ explicitly in terms of $w$ explaining what $x$ is? Dec 3, 2017 at 5:12
• Hi, $y_t \in \{0,1\}$, $x \in \mathbb{R}^n$. They are constants. Dec 3, 2017 at 5:20
• see math.stackexchange.com/questions/1582452/… and let us know if that clarifies your doubts. Dec 3, 2017 at 9:25

For one-dimensional $w$ and $x$, the second derivatives of $-\log (1+a\exp(-wx))$ and of $\log(1-1/(1+a\exp(-wx)))$ w.r.t $w$ both equal $-a x^2 e^{wx}(e^{wx}+a)^{-2}$, where $a>0$. So the function is not convex (for any dimension $n$).
The Hessian matrix for both $-\log(1+\exp(-w\cdot x))$ and $\log(1-1/(1+a\exp(-w\cdot x)))$ is $$-\frac{e^{w\cdot x}}{(1+e^{w\cdot x})^2}x^T x.$$ So the function $f(w)$ is concave.