2
$\begingroup$

The loss function in sklearn is

$$\min_{w,c}{\frac{1}{2}w^Tw+C\sum_{i=1}^N{\log(\exp(-y_i(X_i^Tw+c))+1)}}$$

Whereas the loss function in glmnet is

$$\min_{\beta,\beta_0}{-\bigg[\frac{1}{N} \sum_{i=1}^N{y_i(\beta_0 + x_i^T\beta) - \log(1+e^{\beta_0 + x_i^T\beta})}\bigg] + \lambda[(1-\alpha)||\beta||_2^2/2+\alpha||\beta||_1]}$$

When setting $\alpha=0$, this post states that they differ by a factor of $\lambda$ if $C=\frac{1}{N\lambda}$, but I can't seem to work this out. From the loss function in glmnet and changing the variable names and notations to match that of scikit-learn, I get

$$\begin{align} &\min_{w,c}{-\bigg[\frac{1}{N} \sum_{i=1}^N{y_i(X_i^Tw+c) - \log(1+\exp({X_i^Tw+c}))}\bigg] + \lambda \frac{1}{2}w^Tw} \\ = &\min_{w,c}{\lambda \frac{1}{2}w^Tw -\bigg[\frac{1}{N} \sum_{i=1}^N{y_i(X_i^Tw+c) - \log(1+\exp({X_i^Tw+c}))}\bigg]} \\ = &\min_{w,c}{\lambda \frac{1}{2}w^Tw +\bigg[\frac{1}{N} \sum_{i=1}^N{\log(1+\exp({X_i^Tw+c}))-y_i(X_i^Tw+c)}\bigg]}\\ = &\min_{w,c}{\lambda \frac{1}{2}w^Tw +\bigg[\frac{1}{N} \sum_{i=1}^N{\log(1+\exp({X_i^Tw+c}))-\log(\exp(y_i(X_i^Tw+c)))}\bigg]} \\ = &\min_{w,c}{\lambda \frac{1}{2}w^Tw +\frac{1}{N} \sum_{i=1}^N{\log\bigg(\frac{1+\exp(X_i^Tw+c)}{\exp(y_i(X_i^Tw+c))}\bigg)}} \\ = &\min_{w,c}{\lambda \frac{1}{2}w^Tw +\frac{1}{N} \sum_{i=1}^N{\log\bigg(\frac{1}{\exp(y_i(X_i^Tw+c))} + \frac{\exp(X_i^Tw+c)}{\exp(y_i(X_i^Tw+c))}\bigg)}}\\ = &\min_{w,c}{\lambda \frac{1}{2}w^Tw +\frac{1}{N} \sum_{i=1}^N{\log\bigg(\exp(-y_i(X_i^Tw+c)) + \frac{\exp(X_i^Tw+c)}{\exp(y_i(X_i^Tw+c))}\bigg)}} \end{align}$$

which is only true if $\frac{\exp(X_i^Tw+c)}{\exp(y_i(X_i^Tw+c))}=1$ but I can't reason for that case. Which step is wrong here?

$\endgroup$

1 Answer 1

2
$\begingroup$

Inspired from this answer, I think the reason they are different is that they work on different domains. For sklearn, $y_{sklearn}\in\{-1, 1\}$; for glmnet, $y_{glmnet}\in\{0,1\}$. If we plug in $y=-1$ in the sklearn loss function and $y=0$ in that of glmnet, we get the same result. Same for $y=1$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.