I took Andrew Ng's course "Machine Learning" via Coursera a few months back, not paying attention to most of the math/derivations and instead focusing on implementation and practicality. Since then I have started going back to study some of the underlying theory, and have revisited some of Prof. Ng's lectures. I was reading through his lecture on "Regularized Linear Regression", and saw that he gave the following cost function:

$$J(\theta) = \frac{1}{2m}[\sum_{i=1}^m(h_\theta (x^{(i)}) - y^{(i)})^2 + \lambda\sum_{j=1}^n\theta^2_j]$$

Then, he gives the following gradient for this cost function:

$$\frac{\partial}{\partial \theta_j}J(\theta) = \frac{1}{m}[\sum_{i=1}^m(h_\theta (x^{(i)}) - y^{(i)})x^{(i)}_j - \lambda\theta_j]$$

I am a little confused about how he gets from one to the other. When I tried to do my own derivation, I had the following result:

$$\frac{\partial}{\partial \theta_j}J(\theta) = \frac{1}{m}[\sum_{i=1}^m(h_\theta (x^{(i)}) + y^{(i)})x^{(i)}_j + \lambda\theta_j]$$

The difference is the 'plus' sign between the original cost function and the regularization parameter in Prof. Ng's formula changing into a 'minus' sign in his gradient function, whereas that is not happening in my result.

Intuitively I understand why it's negative: we are reducing the theta parameter by the gradient figure, and we want the regularization parameter to reduce the amount that we are changing the parameter to avoid overfitting. I am just a little stuck on the calculus that backs this intuition.

FYI, you can find the deck here, on slides 15 and 16.

  • 1
    $\begingroup$ In your result you have a "+" preceding the y^(i)--is that a typo? $\endgroup$ – Steve S Aug 10 '14 at 22:26

$J(\theta) = \frac{1}{2m}[\sum_{i=1}^m(h_\theta (x^{(i)}) - y^{(i)})^2 + \lambda\sum_{j=1}^n\theta^2_j]$


$\frac{\partial}{\partial \theta_j}(h_\theta (x^{(i)}) - y^{(i)})^2=2[(h_\theta (x^{(i)}) - y^{(i)})\frac{\partial}{\partial \theta_j}\{h_\theta(x^{(i)})\}]$

Note that in a linear model (being discussed on the pages you mention), $\frac{\partial}{\partial \theta_j}(h_\theta(x^{(i)})=[x^{(i)}]_j$

$\frac{\partial}{\partial \theta_j}\lambda\sum_{j=1}^n\theta^2=2\lambda\theta_j$

So for the linear case

$\frac{\partial}{\partial \theta_j}J(\theta) = \frac{1}{m}[\sum_{i=1}^m(h_\theta (x^{(i)}) - y^{(i)})x^{(i)}_j + \lambda\theta_j]$

Looks like perhaps both you and Andrew might have typos. Well, at least two of the three of us seem to.

  • $\begingroup$ its confirmed, just a typo on the Andrew's note, it should be a + sign. And Prof correctly explain everything correctly including the intuition θ(1-α(λ/m)) meaning every time this shrink θ then minus the usual part before regularisation being introduced. $\endgroup$ – Gob00st Aug 12 '18 at 11:55

Actually if you check the lecture notes just after the video , it shows the formula correctly . The slides that you have lined here shows the exact slide of the video.

enter image description here


Actually, I think that's just a typo.

On slide #16 he writes the derivative of the cost function (with the regularization term) with respect to theta but it's in the context of the Gradient Descent algorithm. Hence, he's also multiplying this derivative by $-\alpha$. Notice: On the second line (of slide 16) he has $-\lambda\theta$ (as you've written), multiplied by $-\alpha$. However, by the third line the multiplied term is still negative even though--if the second line were correct--the negative signs would've cancelled out.

Make sense?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.