1
$\begingroup$

In class we've seen that $a$ (the weights) must satisfy $$X^T (y-Xa) =0$$

Here $X$ is a $(n\times d)$ matrix (so we have $n$ samples in $\mathbb R^d$)

let's denote the residuals $r = y-Xa$. In our class notes, it is written that

The normal equations require the residuals to be orthogonal to each of the columns of $X$.

Why?

Therefore, the solution of the linear regression is a projection of $y$ onto the subspace spanned by $v_1 , \ldots , v_d$ (the columns of $X$)

Can you please explain this?

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
1
$\begingroup$

Recall that the squared error loss you are trying to minimize is $$ (y-Xa)^T (y-Xa) = y^Ty - 2a^TX^Ty + a^TX^TXa; $$ if you take all the derivatives with respect to each element of $a$ and arrange them into a column vector, you get $$ -2X^Ty + 2X^TXa. $$ Then you set that equal to zero to get a necessary condition for the minimization of the sum of squares. These are the "normal equations," and it is the expression you wrote earlier: $$ X^T (y-Xa) =0. $$

When you solve for $a$ you get $\hat{a} = (X^TX)^{-1}X^Ty$, which makes the fitted values (your quote refers to these as "solutions") $$ \hat{y} = X\hat{a} = X(X^TX)^{-1}X^Ty. $$ The matrix $d \times d$ matrix $X(X^TX)^{-1}X^T$ is known as the projection matrix. For more information on why it is called that, see here.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Yeah, alright. so basically $X^T \cdot r = 0$. expanding it, we get: $$r_1 x_1 + \cdots + r_nx_n = 0$$ $\endgroup$
    – deficiencyOn
    Feb 18, 2018 at 18:33
  • $\begingroup$ How can I infer orthogonality? Moreover, the notes says that for every $i,j$: $r_i \cdot x_j = 0$ (if I'm not mistaken) $\endgroup$
    – deficiencyOn
    Feb 18, 2018 at 18:35
  • $\begingroup$ the $\cdot$ is unnecessary to write. But yes, that's right. Orthogonality means that any column vector of $X$, say $x_3$ is orthgonal to $r$, meaning $x_3^Tr = x_3 \cdot r = 0$. The equation you just wrote is writing all $d$ of those equations simultaneously. $\endgroup$
    – Taylor
    Feb 18, 2018 at 18:39
  • $\begingroup$ but how can it be inferred from $X^Tr = 0$? it's probably some linear algebra property, right? $\endgroup$
    – deficiencyOn
    Feb 18, 2018 at 18:46
  • $\begingroup$ @deficiencyOn I proved it for you in my answer. It's from setting the derivatives of the loss function equal to $0$. $\endgroup$
    – Taylor
    Feb 18, 2018 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.