Confusion when going from index notation to matrix notation

Question

In Andrew Ng's lectures on machine learning, on page 4 of his first lecture notes, he presents the cost function for the ordinary least squares regression model as:

$$ \begin{align} J(\theta) = \frac{1}{2} \sum_{i=1}^{m} (\theta^T x^{(i)} - y^{i}))^2 \end{align} $$

He then shows that the partial derivative can be shown as:

$$ \begin{align} \frac{\partial}{\partial \theta_j} J(\theta) = \sum_{i=1}^{m} (\theta^T x^{(i)} - y^{(i)}) x_j^{(i)} \end{align} $$

However in the solution set he gives the gradient of $J(\theta)$ as:

$$ \begin{align} \nabla_\theta J(\theta) = X^TX\theta - X^Ty \end{align} $$

My question is, how do I systematically go from the index form to the matrix form? As of right now, all I am doing is checking the dimensions of the matrices and making sure that they all come out correctly, but this seems to be inadequate. Yet I haven't found a good explanation as to how to convert from one form to the other. Help with this would be greatly appreciated.

Answer 1

All vector are columns vectors if not transposed. We have $$X = (x_1, \dots, x_p) = \begin{pmatrix} (x^{(1)})^T \\ (x^{(2)})^T \\ \vdots \\ (x^{(n)})^T \\ \end{pmatrix} \in \Re^{n \times p}$$. Now we can rewrite the sum as a dot product

$$\frac{\partial}{\partial\theta_j}J(\theta) = \sum_{i=1}^{n}(\theta^Tx^{(i)} - y^{(i)})x_j^{(i)} = w^Tx_j = x_j^Tw$$

For some vector $w$. Now

$$ w = \begin{pmatrix} \theta^Tx^{(1)} - y^{(1)} \\ \theta^Tx^{(2)} - y^{(2)} \\ \vdots \\ \theta^Tx^{(n)} - y^{(n)} \\ \end{pmatrix} = \begin{pmatrix} \theta^Tx^{(1)}\\ \theta^Tx^{(2)}\\ \vdots \\ \theta^Tx^{(n)}\\ \end{pmatrix} - \begin{pmatrix} y^{(1)} \\ y^{(2)} \\ \vdots \\ y^{(n)} \\ \end{pmatrix} = \begin{pmatrix} (x^{(1)})^T\theta\\ (x^{(2)})^T\theta\\ \vdots \\ (x^{(n)})^T\theta\\ \end{pmatrix} - Y = X\theta - Y $$ So we have shown that $$\frac{\partial}{\partial\theta_j}J(\theta) = x_j^Tw = x_j^T(X\theta - Y)$$ So in the end $$ \nabla_{\theta}J(\theta) = \begin{pmatrix} x_1^T(X\theta - Y) \\ x_2^T(X\theta - Y) \\ \vdots \\ x_p^T(X\theta - Y) \\ \end{pmatrix} = X^T(X\theta - Y) = X^TX\theta - X^TY $$