1
$\begingroup$

In Andrew Ng's lectures on machine learning, on page 4 of his first lecture notes, he presents the cost function for the ordinary least squares regression model as:

$$ \begin{align} J(\theta) = \frac{1}{2} \sum_{i=1}^{m} (\theta^T x^{(i)} - y^{i}))^2 \end{align} $$

He then shows that the partial derivative can be shown as:

$$ \begin{align} \frac{\partial}{\partial \theta_j} J(\theta) = \sum_{i=1}^{m} (\theta^T x^{(i)} - y^{(i)}) x_j^{(i)} \end{align} $$

However in the solution set he gives the gradient of $J(\theta)$ as:

$$ \begin{align} \nabla_\theta J(\theta) = X^TX\theta - X^Ty \end{align} $$

My question is, how do I systematically go from the index form to the matrix form? As of right now, all I am doing is checking the dimensions of the matrices and making sure that they all come out correctly, but this seems to be inadequate. Yet I haven't found a good explanation as to how to convert from one form to the other. Help with this would be greatly appreciated.

$\endgroup$
  • $\begingroup$ You might find it easier to go in the other direction: use the definitions of matrix multiplication and the gradient to convert the matrix formula into one involving the indexes. This is purely mechanical (it's not hard to write a program to do it). After a little practice you will find you can easily go in the other direction, too. $\endgroup$ – whuber Mar 11 '17 at 19:58
  • $\begingroup$ math.stackexchange.com/questions/198257/… $\endgroup$ – kjetil b halvorsen Mar 13 '17 at 18:31
2
$\begingroup$

All vector are columns vectors if not transposed. We have $$X = (x_1, \dots, x_p) = \begin{pmatrix} (x^{(1)})^T \\ (x^{(2)})^T \\ \vdots \\ (x^{(n)})^T \\ \end{pmatrix} \in \Re^{n \times p}$$. Now we can rewrite the sum as a dot product

$$\frac{\partial}{\partial\theta_j}J(\theta) = \sum_{i=1}^{n}(\theta^Tx^{(i)} - y^{(i)})x_j^{(i)} = w^Tx_j = x_j^Tw$$

For some vector $w$. Now

$$ w = \begin{pmatrix} \theta^Tx^{(1)} - y^{(1)} \\ \theta^Tx^{(2)} - y^{(2)} \\ \vdots \\ \theta^Tx^{(n)} - y^{(n)} \\ \end{pmatrix} = \begin{pmatrix} \theta^Tx^{(1)}\\ \theta^Tx^{(2)}\\ \vdots \\ \theta^Tx^{(n)}\\ \end{pmatrix} - \begin{pmatrix} y^{(1)} \\ y^{(2)} \\ \vdots \\ y^{(n)} \\ \end{pmatrix} = \begin{pmatrix} (x^{(1)})^T\theta\\ (x^{(2)})^T\theta\\ \vdots \\ (x^{(n)})^T\theta\\ \end{pmatrix} - Y = X\theta - Y $$ So we have shown that $$\frac{\partial}{\partial\theta_j}J(\theta) = x_j^Tw = x_j^T(X\theta - Y)$$ So in the end $$ \nabla_{\theta}J(\theta) = \begin{pmatrix} x_1^T(X\theta - Y) \\ x_2^T(X\theta - Y) \\ \vdots \\ x_p^T(X\theta - Y) \\ \end{pmatrix} = X^T(X\theta - Y) = X^TX\theta - X^TY $$

$\endgroup$
  • $\begingroup$ I see that this is the case just out of familiarity with linear algebraic notation; however, I have to say that I wouldn't be able to follow your answer if I was trying to understand this for the first time. It's not that your answer is wrong, but it is cut and dry in its pure mathematical notation with no English cushioning explanations. $\endgroup$ – Antoni Parellada Mar 11 '17 at 18:45
  • $\begingroup$ @AntoniParellada Yeah you're probably right. Feel free to add anything that will improve friendliness now that all the math is here already : ) $\endgroup$ – Łukasz Grad Mar 11 '17 at 18:59
  • $\begingroup$ This would be completely against etiquette. I would write my own answer, but I think your answer is all that is needed. My suggestion was to make it more accessible, but it's up to you. $\endgroup$ – Antoni Parellada Mar 11 '17 at 19:05
  • $\begingroup$ @AntoniParellada, perhaps it might be against etiquette, but given that Lukasz doesn't mind, perhaps you could go ahead and make it more accessible, at least for me? =) $\endgroup$ – oort Mar 11 '17 at 19:39
  • $\begingroup$ @oort It's not that I don't want to do it myself, if you can specify which step is not clear I will try to explain in more detail $\endgroup$ – Łukasz Grad Mar 11 '17 at 19:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.