6
$\begingroup$

I am having trouble to understand the loss function scikit-learn uses to fit logistic regression, which can be found here.

Specifically I have problem with the second term. It seems very different from the usual MLE criterion. Can someone give me some hint where this comes from?

$$\mathop {\min{\mkern 1mu} }\limits_{w,c} \frac{1}{2}{w^T}w + C\sum\limits_{i = 1}^n {\log } (\exp ( - {y_i}(X_i^Tw + c)) + 1)$$

I think usually the log likelihood of a logistic regression is something like below. Clearly the first term of below is missing from the scikit-learn objective function.

$$LLH=\sum_{i=1}^n \left[{y_i}(X_i^Tw + c) - \ln\{1+\exp(X_i^Tw + c)\} \right]$$

$\endgroup$
4
$\begingroup$

These two are actually (almost) equivalent because of the following property of the logistic function:

$$ \sigma(x) = \frac{1}{1+\exp(-x)} = \frac{\exp(x)}{\exp(x)+1} $$

Also

$$ \sum_{i=1}^n \log ( 1 + \exp( -y_i (X_i^T w + c) ) ) \\ = \sum_{i=1}^n \log \left[ (\exp( y_i (X_i^T w + c) ) + 1) \exp( -y_i (X_i^T w + c) ) \right] \\ = -\sum_{i=1}^n \left[ y_i (X_i^T w + c) - \log (\exp( y_i (X_i^T w + c) ) + 1) \right] $$

Note, though, that your formula doesn't have $y_i$ in the "log part", while this one does. (I guess this is a typo)

$\endgroup$
2
$\begingroup$

I don't think that the lack of $y_i$ is a typo:

The usual log-loss (cross-entropy loss) is: $$-\sum_i [y_i \log(p_i) + (1-y_i) \log(1 - p_i)],$$ where $p_i = \sigma(X^T_i \omega + c)$, and $\sigma(x) = 1/(1+e^{-x})$ is the logistic function.

From there, $$-\sum_i [y_i \log(p_i) + (1-y_i) \log(1 - p_i)] \\ = -\sum_i [y_i \log\left(\frac{p_i}{1-p_i}\right) + \log(1 - p_i)] \\ = -\sum_i [y_i \left( X^T_i \omega + c \right) + \log(1 - p_i)] \\ = -\sum_i [y_i \left( X^T_i \omega + c \right) - \log\left(1 + \exp({X^T_i \omega + c})\right)].$$ This matches the LLH expression given in the original post, without the $y_i$ factor in the exponential.

$\endgroup$
1
$\begingroup$

It is just a matter of the definition of $y_i$. Defining $y_i$ and $\tilde y_i$ such that $y_i \in \{0, 1\}$ and $\tilde y_i \in \{-1, 1\}$ ($\tilde y_i = 2y_i -1$), and using $p_i = \sigma({X^T_i \omega + c})$ and $1- \sigma(x) = \sigma(-x)$, you get

$$-\sum_i [y_i \log(p_i) + (1-y_i) \log(1 - p_i)] = \sum_i \log\left(1 + \exp(-\tilde y_i({X^T_i \omega + c}))\right).$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.