Confusion when going from index notation to matrix notation In Andrew Ng's lectures on machine learning, on page 4 of his first lecture notes, he presents the cost function for the ordinary least squares regression model as:
$$
\begin{align}
J(\theta) = \frac{1}{2} \sum_{i=1}^{m} (\theta^T x^{(i)} - y^{i}))^2
\end{align}
$$
He then shows that the partial derivative can be shown as:
$$
\begin{align}
\frac{\partial}{\partial \theta_j} J(\theta) = \sum_{i=1}^{m} (\theta^T x^{(i)} - y^{(i)}) x_j^{(i)}
\end{align}
$$
However in the solution set he gives the gradient of $J(\theta)$ as:
$$
\begin{align}
\nabla_\theta J(\theta) = X^TX\theta - X^Ty
\end{align}
$$
My question is, how do I systematically go from the index form to the matrix form? As of right now, all I am doing is checking the dimensions of the matrices and making sure that they all come out correctly, but this seems to be inadequate. Yet I haven't found a good explanation as to how to convert from one form to the other. Help with this would be greatly appreciated.
 A: All vector are columns vectors if not transposed. We have 
$$X = (x_1, \dots, x_p) = \begin{pmatrix}
(x^{(1)})^T \\
(x^{(2)})^T \\
\vdots \\
(x^{(n)})^T \\
\end{pmatrix}  \in \Re^{n \times p}$$. Now we can rewrite the sum as a dot product
$$\frac{\partial}{\partial\theta_j}J(\theta) = \sum_{i=1}^{n}(\theta^Tx^{(i)} - y^{(i)})x_j^{(i)} = w^Tx_j = x_j^Tw$$
For some vector $w$. Now
$$
w = 
\begin{pmatrix}
\theta^Tx^{(1)} - y^{(1)} \\
\theta^Tx^{(2)} - y^{(2)} \\
\vdots \\
\theta^Tx^{(n)} - y^{(n)} \\
\end{pmatrix}
=
\begin{pmatrix}
\theta^Tx^{(1)}\\
\theta^Tx^{(2)}\\
\vdots \\
\theta^Tx^{(n)}\\
\end{pmatrix}
-
\begin{pmatrix}
y^{(1)} \\
y^{(2)} \\
\vdots \\
y^{(n)} \\
\end{pmatrix}
=
\begin{pmatrix}
(x^{(1)})^T\theta\\
(x^{(2)})^T\theta\\
\vdots \\
(x^{(n)})^T\theta\\
\end{pmatrix}
-
Y
=
X\theta - Y
$$
So we have shown that 
$$\frac{\partial}{\partial\theta_j}J(\theta) = x_j^Tw = x_j^T(X\theta - Y)$$
So in the end
$$
\nabla_{\theta}J(\theta) = 
\begin{pmatrix}
x_1^T(X\theta - Y) \\
x_2^T(X\theta - Y) \\
\vdots \\
x_p^T(X\theta - Y) \\
\end{pmatrix}
=
X^T(X\theta - Y) = X^TX\theta - X^TY
$$
