8
$\begingroup$

The log likelihood function of logistic regression is as below:

\begin{align} \ln(L(x, y; w)) &=\sum\limits_{i=1}^n [y_i\ln(p(x_i;w))+(1-y_i)\ln(1-p(x_i;w))] \\&=\sum\limits_{i=1}^n [y_i\ln(\dfrac{1}{1+e^{-w'x_i}})+(1-y_i)\ln(1-\dfrac{1}{1+e^{-w'x_i}})] \end{align}


The cost function including penalty and L1/L2 regularization is as below, see link enter image description here


I understand C and L1/L2 norm but cannot derive cost function, can anyone help with the derivation process?

$\endgroup$
2
  • $\begingroup$ It isn't clear to me what you mean by "cannot derive to cost function" - is it just that you do not know to convert the equation into Python code? That would be off-topic here - see our help center - but it is possible you have an underlying statistical issue here which would be on-topic. Can you clarify what you're trying to achieve, and what you would hope to learn from an answer to your question? $\endgroup$
    – Silverfish
    Sep 17 '16 at 20:10
  • 1
    $\begingroup$ I mean mathematically I don't know how to reach to the cost function from my likelihood function. Not how to use code in Python. Basically, I'd like to know the mathematical derivation part @Silverfish $\endgroup$
    – Nicholas
    Sep 17 '16 at 20:14
12
$\begingroup$

Your log-likelihood is: $$ \log L(x, y; w) = \sum_{i=1}^N \ell_i $$ where \begin{align} \ell_i &= y_i \log\left( \frac{1}{1 + \exp(- w^T x_i)} \right) + (1-y_i) \log\left( 1 - \frac{1}{1 + \exp(- w^T x_i)} \right) \\&= y_i \log\left( \frac{1}{1 + \exp(- w^T x_i)} \right) + (1-y_i) \log\left( \frac{1 + \exp(- w^T x_i)}{1 + \exp(- w^T x_i)} - \frac{1}{1 + \exp(- w^T x_i)} \right) \\&= y_i \log\left( \frac{1}{1 + \exp(- w^T x_i)} \right) + (1-y_i) \log\left( \frac{\exp(- w^T x_i)}{1 + \exp(- w^T x_i)} \right) \\&= y_i \log\left( \frac{1}{1 + \exp(- w^T x_i)} \right) + (1-y_i) \log\left( \frac{\exp(- w^T x_i)}{1 + \exp(- w^T x_i)} \times \frac{\exp(w^T x_i)}{\exp(w^T x_i)} \right) \\&= y_i \log\left( \frac{1}{1 + \exp(- w^T x_i)} \right) + (1-y_i) \log\left( \frac{1}{\exp(w^T x_i) + 1} \right) \\&= \log\left( \frac{1}{1 + \exp\left( \begin{cases}- w^T x_i & y_i = 1 \\ w^T x_i & y_i = 0\end{cases} \right)} \right) \\&= \log\left( \frac{1}{1 + \exp\left( - y'_i w^T x_i \right)} \right) \\&= -\log\left( 1 + \exp\left( - y_i' w^T x_i \right) \right) \end{align} where $y_i \in \{0, 1\}$ but we defined $y_i' \in \{-1, 1\}$.

To get to the loss function in the image, first we need to add an intercept to the model, replacing $w^T x_i$ with $w^T x_i + c$. Then: $$ \arg\max \log L(X, y; w, c) = \arg\min - \log L(X, y; w, c) ,$$ and then we add a regularizer $P(c, w)$: $$ \arg\min \lambda P(w, c) - \log L(X, y; w, c) = \arg\min P(w, c) - \frac{1}{\lambda} \log L(X, y; w, c) ,$$ where we then set $C := \frac1\lambda$. The $L_2$ penalty is $$ P(w, c) = \frac12 w^T w = \frac12 \sum_{j=1}^d w_j^2 ;$$ that $\tfrac12$ is just done for mathematical convenience when we differentiate, it doesn't really affect anything. The $L_1$ penalty has $$ P(w, c) = \lVert w \rVert_1 = \sum_{j=1}^d \lvert w_j \rvert .$$

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.