I am doing the Machine Learning Stanford course on Coursera.

In the chapter on Logistic Regression, the cost function is this: enter image description here

Then, it is derivated here: enter image description here

I tried getting the derivative of the cost function but I got something completely different.

How is the derivative obtained?

Which are the intermediary steps?

  • +1, check @AdamO's answer in my question here. stats.stackexchange.com/questions/229014/… – hxd1011 May 10 '17 at 18:39
  • "Completely different" is not really sufficient to answer your question, besides telling you what you already know (the correct gradient). It'd be much more useful if you gave us what your calculations resulted in, then we can help you shore up where you made the mistake. – Matthew Drury May 10 '17 at 20:43
  • @MatthewDrury Sorry, Matt, I had arranged the answer right before your comment came in. Octavian, did you follow all the steps? I will edit to give it some added value later... – Antoni Parellada May 10 '17 at 20:46
  • 1
    when you say "derivated" do you mean "differentiated" or "derived"? – Glen_b May 11 '17 at 3:19
up vote 24 down vote accepted

Adapted from the notes in the course, which I don't see available (including this derivation) outside the notes contributed by students within the page of Andrew Ng's Coursera Machine Learning course.


In what follows, the superscript $(i)$ denotes individual measurements or training "examples."

$$\begin{align} \frac{\partial J(\theta)}{\partial \theta_j} &= \frac{\partial}{\partial \theta_j} \,\frac{-1}{m}\sum_{i=1}^m \left[ y^{(i)}\left(\log(h_\theta \left(x^{(i)}\right)\right) + (1 -y^{(i)})\left(\log(1-h_\theta \left(x^{(i)}\right)\right)\right]\\[2ex] &\underset{\text{linearity}}= \,\frac{-1}{m}\,\sum_{i=1}^m \left[ y^{(i)}\frac{\partial}{\partial \theta_j}\log\left(h_\theta \left(x^{(i)}\right)\right) + (1 -y^{(i)})\frac{\partial}{\partial \theta_j}\left(\log(1-h_\theta \left(x^{(i)}\right)\right) \right]\\[2ex] &\underset{\text{chain rule}}= \,\frac{-1}{m}\,\sum_{i=1}^m \left[ y^{(i)}\frac{\frac{\partial}{\partial \theta_j}(h_\theta \left(x^{(i)}\right)}{h_\theta\left(x^{(i)}\right)} + (1 -y^{(i)})\frac{\frac{\partial}{\partial \theta_j}\left(1-h_\theta \left(x^{(i)}\right)\right)}{1-h_\theta\left(x^{(i)}\right)} \right]\\[2ex] &\underset{h_\theta(x)=\sigma\left(\theta^\top x\right)}=\,\frac{-1}{m}\,\sum_{i=1}^m \left[ y^{(i)}\frac{\frac{\partial}{\partial \theta_j}\sigma\left(\theta^\top x^{(i)}\right)}{h_\theta\left(x^{(i)}\right)} + (1 -y^{(i)})\frac{\frac{\partial}{\partial \theta_j}\left(1-\sigma\left(\theta^\top x^{(i)}\right)\right)}{1-h_\theta\left(x^{(i)}\right)} \right]\\[2ex] &\underset{\sigma'}=\frac{-1}{m}\,\sum_{i=1}^m \left[ y^{(i)}\, \frac{\sigma\left(\theta^\top x^{(i)}\right)\left(1-\sigma\left(\theta^\top x^{(i)}\right)\right)\frac{\partial}{\partial \theta_j}\left(\theta^\top x^{(i)}\right)}{h_\theta\left(x^{(i)}\right)}\\ - (1 -y^{(i)})\,\frac{\sigma\left(\theta^\top x^{(i)}\right)\left(1-\sigma\left(\theta^\top x^{(i)}\right)\right)\frac{\partial}{\partial \theta_j}\left(\theta^\top x^{(i)}\right)}{1-h_\theta\left(x^{(i)}\right)} \right]\\[2ex] &\underset{\sigma\left(\theta^\top x\right)=h_\theta(x)}= \,\frac{-1}{m}\,\sum_{i=1}^m \left[ y^{(i)}\frac{h_\theta\left( x^{(i)}\right)\left(1-h_\theta\left( x^{(i)}\right)\right)\frac{\partial}{\partial \theta_j}\left(\theta^\top x^{(i)}\right)}{h_\theta\left(x^{(i)}\right)} \\- (1 -y^{(i)})\frac{h_\theta\left( x^{(i)}\right)\left(1-h_\theta\left(x^{(i)}\right)\right)\frac{\partial}{\partial \theta_j}\left( \theta^\top x^{(i)}\right)}{1-h_\theta\left(x^{(i)}\right)} \right]\\[2ex] &\underset{\frac{\partial}{\partial \theta_j}\left(\theta^\top x^{(i)}\right)=x_j^{(i)}}=\,\frac{-1}{m}\,\sum_{i=1}^m \left[y^{(i)}\left(1-h_\theta\left(x^{(i)}\right)\right)x_j^{(i)}- \left(1-y^{i}\right)\,h_\theta\left(x^{(i)}\right)x_j^{(i)} \right]\\[2ex] &\underset{\text{distribute}}=\,\frac{-1}{m}\,\sum_{i=1}^m \left[y^{i}-y^{i}h_\theta\left(x^{(i)}\right)- h_\theta\left(x^{(i)}\right)+y^{(i)}h_\theta\left(x^{(i)}\right) \right]\,x_j^{(i)}\\[2ex] &\underset{\text{cancel}}=\,\frac{-1}{m}\,\sum_{i=1}^m \left[y^{(i)}-h_\theta\left(x^{(i)}\right)\right]\,x_j^{(i)}\\[2ex] &=\frac{1}{m}\sum_{i=1}^m\left[h_\theta\left(x^{(i)}\right)-y^{(i)}\right]\,x_j^{(i)} \end{align}$$


The derivative of the sigmoid function is

$\begin{align}\frac{d}{dx}\sigma(x)&=\frac{d}{dx}\left(\frac{1}{1+e^{-x}}\right)\\[2ex] &=\frac{-(1+e^{-x})'}{(1+e^{-x})^2}\\[2ex] &=\frac{e^{-x}}{(1+e^{-x})^2}\\[2ex] &=\left(\frac{1}{1+e^{-x}}\right)\left(\frac{e^{-x}}{1+e^{-x}}\right)\\[2ex] &=\left(\frac{1}{1+e^{-x}}\right)\,\left(\frac{1+e^{-x}}{1+e^{-x}}-\frac{1}{1+e^{-x}}\right)\\[2ex] &=\sigma(x)\,\left(\frac{1+e^{-x}}{1+e^{-x}}-\sigma(x)\right)\\[2ex] &=\sigma(x)\,(1-\sigma(x)) \end{align}$

  • +1 for all the efforts!, may be using matrix notation could be easier? – hxd1011 May 11 '17 at 1:43
  • can I say in linear regression, objective is $\|Ax-b\|^2$ and derivative is $2A^Te$, where $e=Ax-b$, in logistic regression, it is similar, the derivative is $A^Te$ where $e=p-b$, and $p=\text{sigmoid}~(Ax)$ ? – hxd1011 May 11 '17 at 1:46
  • @hxd1011 Ty! Yes, no more $\sum$'s, but wanted to be consistent with OP. – Antoni Parellada May 11 '17 at 1:46
  • 1
    that is why I appreciate your effort. you spend time to us OP's language!! – hxd1011 May 11 '17 at 1:47
  • My understanding is that there are convexity issues that make the squared error minimization undesirable for non-linear activation functions. In matrix notation, it would be $\frac{\partial J(\theta)}{\partial \theta}=\frac{1}{m}X^\top\left( \sigma(X\theta)-\mathbf y\right)$. – Antoni Parellada May 11 '17 at 1:57

To avoid impression of excessive complexity of the matter, let us just see the structure of solution.

With simplification and some abuse of notation, let $G(\theta)$ be a term in sum of $J(\theta)$, and $h = 1/(1+e^{-z})$ is a function of $z(\theta)= x \theta $: $$ G = y \cdot \log(h)+(1-y)\cdot \log(1-h) $$

We may use chain rule: $\frac{d G}{d \theta}=\frac{d G}{d h}\frac{d h}{d z}\frac{d z}{d \theta}$ and solve it one by one ($x$ and $y$ are constants).

$$\frac{d G}{\partial h} = \frac{y} {h} - \frac{1-y}{1-h} = \frac{y - h}{h(1-h)} $$ For sigmoid $\frac{d h}{d z} = h (1-h) $ holds, which is just a denominator of the previous statement.

Finally, $\frac{d z}{d \theta} = x $.

Combining results all together gives sought-for expression: $$\frac{d G}{d \theta} = (y-h)x $$ Hope that helps.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.