Understanding linear regression

In class we've seen that $a$ (the weights) must satisfy $$X^T (y-Xa) =0$$

Here $X$ is a $(n\times d)$ matrix (so we have $n$ samples in $\mathbb R^d$)

let's denote the residuals $r = y-Xa$. In our class notes, it is written that

The normal equations require the residuals to be orthogonal to each of the columns of $X$.

Why?

Therefore, the solution of the linear regression is a projection of $y$ onto the subspace spanned by $v_1 , \ldots , v_d$ (the columns of $X$)

Recall that the squared error loss you are trying to minimize is $$(y-Xa)^T (y-Xa) = y^Ty - 2a^TX^Ty + a^TX^TXa;$$ if you take all the derivatives with respect to each element of $a$ and arrange them into a column vector, you get $$-2X^Ty + 2X^TXa.$$ Then you set that equal to zero to get a necessary condition for the minimization of the sum of squares. These are the "normal equations," and it is the expression you wrote earlier: $$X^T (y-Xa) =0.$$
When you solve for $a$ you get $\hat{a} = (X^TX)^{-1}X^Ty$, which makes the fitted values (your quote refers to these as "solutions") $$\hat{y} = X\hat{a} = X(X^TX)^{-1}X^Ty.$$ The matrix $d \times d$ matrix $X(X^TX)^{-1}X^T$ is known as the projection matrix. For more information on why it is called that, see here.
• Yeah, alright. so basically $X^T \cdot r = 0$. expanding it, we get: $$r_1 x_1 + \cdots + r_nx_n = 0$$ Feb 18, 2018 at 18:33
• How can I infer orthogonality? Moreover, the notes says that for every $i,j$: $r_i \cdot x_j = 0$ (if I'm not mistaken) Feb 18, 2018 at 18:35
• the $\cdot$ is unnecessary to write. But yes, that's right. Orthogonality means that any column vector of $X$, say $x_3$ is orthgonal to $r$, meaning $x_3^Tr = x_3 \cdot r = 0$. The equation you just wrote is writing all $d$ of those equations simultaneously. Feb 18, 2018 at 18:39
• but how can it be inferred from $X^Tr = 0$? it's probably some linear algebra property, right? Feb 18, 2018 at 18:46
• @deficiencyOn I proved it for you in my answer. It's from setting the derivatives of the loss function equal to $0$. Feb 18, 2018 at 18:56