I would like to know why $ \beta = (X^T X)^{-1} X^T y $ is the solution for the ML-estimator for the linear regression.
In principle it is shown, e.g., here in this file respectively in specific I would like to know where the inverse comes from?
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ The answer is given here although the question is different. Essentially, you just have to show that $\hat{\beta}$ achieves your objective (either minimizing square error or maximizing likelihood, which are equivalent in this case). stats.stackexchange.com/questions/286497/… $\endgroup$– Sycorax ♦Aug 28, 2019 at 12:55
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
in the notes you use, $X$ is a $T \times k$ matrix, to find the extrema one needs to set the right-hand side of Eq. 7 to zero, which is equivalent to solving
$(y - X \beta)' = 0 $
as it should vanish for every $X$. Note on the previous equation $y$, and $\beta$ are vectors while $X$ is a matrix, the zero is actually a $T$-dimensional vector of zeroes.
This eq is a linear system, but the matrix $X$ is not invertible in general, so you cannot solve it as $\beta = X^{-1} y$ since $ X^{-1}$ would not be defined. You can do a trick to bypass this, multiply each side by $X^T$ (the transpose) and then you have
$( X^T y - (X^T X ) \beta)' = 0 $
Now $X^T X$ is a positive definite symmetric matrix so we know it is invertible, now you can use the usual solution for a linear system
$\beta = (X^T X )^{-1} X^T y $
which is what you have.
-
4$\begingroup$ Good answer, I just suggest an edit: $X_{T}X$ is a positive definite matrix, not a positive semidefinite. Otherwise it would be a singular matrix, and, as such, no inverse would be available and no beta. In other words, if there is perfect multicollinearity between the regressors, then no beta exists. $\endgroup$– Fr1Aug 28, 2019 at 9:19
-