1
$\begingroup$

Kaggle defines the log loss function as: https://www.kaggle.com/wiki/LogarithmicLoss

$$f(w) = \log \Pr(y_t|y_p) = y_t \log(y_p) + (1 - y_t) \log(1 - y_p)$$

Let $y_t \in {0, 1}$, and $y_p$ is given by the sigmoid function $y_p = \sigma(w \cdot x) \in (0,1)$, where $\sigma$ is defined by

$$\sigma(w \cdot x) = \dfrac{1}{(1+\exp(-w \cdot x)}$$

$w,x \in \mathbb{R}^n$. The objective variable is $w$ so $x$ could be thought of as a weight.

So putting everything together:

$$f(w) = y_t \log\left(\dfrac{1}{(1+\exp(-w \cdot x)}\right) + (1 - y_t) \log\left(1 - \dfrac{1}{(1+\exp(-w \cdot x)}\right)$$

My question is: Is $f(w)$ convex?

I have read up some references but can't see the relationship: http://qwone.com/~jason/writing/convexLR.pdf. If anyone has a reference, it will be greatly appreciated!!

$\endgroup$
3
  • $\begingroup$ And what is $y_t$? Could you rewrite $f(w)$ explicitly in terms of $w$ explaining what $x$ is? $\endgroup$
    – Viktor
    Dec 3, 2017 at 5:12
  • $\begingroup$ Hi, $y_t \in \{0,1\}$, $x \in \mathbb{R}^n$. They are constants. $\endgroup$
    – Fraïssé
    Dec 3, 2017 at 5:20
  • $\begingroup$ see math.stackexchange.com/questions/1582452/… and let us know if that clarifies your doubts. $\endgroup$
    – DeltaIV
    Dec 3, 2017 at 9:25

1 Answer 1

3
$\begingroup$

For one-dimensional $w$ and $x$, the second derivatives of $-\log (1+a\exp(-wx))$ and of $\log(1-1/(1+a\exp(-wx)))$ w.r.t $w$ both equal $-a x^2 e^{wx}(e^{wx}+a)^{-2}$, where $a>0$. So the function is not convex (for any dimension $n$).

However negative second partial derivatives give us a hint that the function is probably concave. We may consider the Hessian matrix to prove it.

The Hessian matrix for both $-\log(1+\exp(-w\cdot x))$ and $\log(1-1/(1+a\exp(-w\cdot x)))$ is $$-\frac{e^{w\cdot x}}{(1+e^{w\cdot x})^2}x^T x. $$ So the function $f(w)$ is concave.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.