1
$\begingroup$

When dealing with regression, specifically the normal equation, it is derived via calculus, as it is here:

https://eli.thegreenplace.net/2014/derivation-of-the-normal-equation-for-linear-regression

But, I don't see where you even need calculus. Why not just do the following using basic matrix operations (assuming that $X^T X$ is invertible): $$ X\beta = y \\ X^T \cdot X\beta = X^T\cdot y \\ (X^T X)^{-1}\cdot X^T X\beta = (X^T X)^{-1}\cdot X^T y \\ \beta = (X^T X)^{-1}\cdot X^Ty \\ $$

Is there anything wrong with this?

One problem I have with this is: doesn't this imply that the resulting $\beta $ will make it so that the plane actually interpolates every point of $y$. This is often not the case when doing linear regression, so this is what seems weird.

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ Yes, the first assumption, that $X\beta = y$, is wrong. In Linear regression, the assumption is $y = X\beta + \epsilon$ $\endgroup$
    – Tim Mak
    Commented May 14, 2020 at 1:49
  • $\begingroup$ @TimMak Hmmm, what about if you collect data and are only looking at the realizations (ie data). Then you would have a vector $y$ and a matrix $X$ and the $\epsilon$ are not really present. Then your objective is to solve for the vector $\beta$ . $\endgroup$
    – Jac Frall
    Commented May 14, 2020 at 2:06
  • 1
    $\begingroup$ $\epsilon$ are present. You don't have $X\beta=y$, you have $X\beta+\epsilon=y$, as Tim said. So your method for solving isn't applicable here since it's based off an equation that isn't valid. $\endgroup$
    – norvia
    Commented May 14, 2020 at 2:09
  • $\begingroup$ Thanks everyone, after researching some more I finally stumbled upon what was causing the confusion math.stackexchange.com/questions/2577065/… $\endgroup$
    – Jac Frall
    Commented May 14, 2020 at 2:36
  • 1
    $\begingroup$ Then, if you solved your problem, you can answer your own question here (in the answer box) so it does not linger on as unresolved. $\endgroup$
    – kjetil b halvorsen ♦
    Commented May 14, 2020 at 4:04

1 Answer 1

Reset to default
1
$\begingroup$

The thing that was messing me up was something from linear algebra, or just regular algebra really.

The statement $$a=b \quad\Longrightarrow\quad Ta=Tb$$ is true, but the converse $$Ta=Tb \quad\Longrightarrow\quad a=b$$ is not always true.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.