2
$\begingroup$

I'm trying to understand the derivation of the equations for the logistic regression. I'm following the cs229 notes:

http://cs229.stanford.edu/notes/cs229-notes1.pdf

At some point in the derivation, in the notes we need to maximize the log-likelihood of the parameters:

ℓ(θ) = log L(θ) = y log h(x) + (1 - y) log(1-h(x))

In the notes, after the derivation we get:

$\frac{\partial}{\partial \theta_j}$ = (y $\frac 1 {h(x)}$ - (1 - y) $\frac 1 {1 - h(x)}$) $\frac{\partial}{\partial \theta_j}$h(x)

Shouldn't this be:

$\frac{\partial}{\partial \theta_j}$ = (y $\frac 1 {h(x)}$ + (1 - y) $\frac 1 {1 - h(x)}$) $\frac{\partial}{\partial \theta_j}$h(x)

I've been looking at this for a while and I can't see where this minus sign is coming from...

Thanks for your help!

$\endgroup$
2
  • $\begingroup$ $\frac{\partial \log (1-h(x,\theta))}{\partial\theta}=\frac{1}{1-h(x,\theta)}\frac{\partial (1-h(x, \theta) ) }{\partial\theta}=\frac{1}{1-h(x,\theta)}(-1)\frac{\partial h}{\partial\theta}$. For more details I refer to math.stackexchange.com/questions/477207/… $\endgroup$
    – Avitus
    Dec 23, 2014 at 11:44
  • $\begingroup$ There is a mistake in the different equations in that there is not $\theta$. Do you mean $h(x^\text{T}\theta)$? $\endgroup$
    – Xi'an
    Dec 23, 2014 at 14:30

1 Answer 1

3
$\begingroup$

There is little to explain: \begin{align*} \frac{\partial}{\partial \theta_j} \ell(\theta) &= \frac{\partial}{\partial \theta_j} \left\{ y \log h(x) + (1 - y) \log(1-h(x)) \right\}\\ &= y \frac{\partial}{\partial \theta_j} \log h(x) + (1 - y) \frac{\partial}{\partial \theta_j} \log(1-h(x))\\ &= y \frac{1}{h(x)} \frac{\partial}{\partial \theta_j} h(x) + (1 - y) \frac{-1}{1-h(x)}\frac{\partial}{\partial \theta_j} h(x)\\ &= \left( \frac{y}{h(x)}-\frac{1-y}{1-h(x)} \right) \frac{\partial h(x)}{\partial \theta_j} \end{align*} So it is because $$\frac{\text{d}}{\text{d}x}\log(1-x) = \frac{-1}{1-x}$$ that you get this expression.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.