# MLE estimator linear regression: Why inverse? [duplicate]

I would like to know why $$\beta = (X^T X)^{-1} X^T y$$ is the solution for the ML-estimator for the linear regression.

In principle it is shown, e.g., here in this file respectively in specific I would like to know where the inverse comes from?

• The answer is given here although the question is different. Essentially, you just have to show that $\hat{\beta}$ achieves your objective (either minimizing square error or maximizing likelihood, which are equivalent in this case). stats.stackexchange.com/questions/286497/…
– Sycorax
Aug 28, 2019 at 12:55

in the notes you use, $$X$$ is a $$T \times k$$ matrix, to find the extrema one needs to set the right-hand side of Eq. 7 to zero, which is equivalent to solving

$$(y - X \beta)' = 0$$

as it should vanish for every $$X$$. Note on the previous equation $$y$$, and $$\beta$$ are vectors while $$X$$ is a matrix, the zero is actually a $$T$$-dimensional vector of zeroes.

This eq is a linear system, but the matrix $$X$$ is not invertible in general, so you cannot solve it as $$\beta = X^{-1} y$$ since $$X^{-1}$$ would not be defined. You can do a trick to bypass this, multiply each side by $$X^T$$ (the transpose) and then you have

$$( X^T y - (X^T X ) \beta)' = 0$$

Now $$X^T X$$ is a positive definite symmetric matrix so we know it is invertible, now you can use the usual solution for a linear system

$$\beta = (X^T X )^{-1} X^T y$$

which is what you have.

• Good answer, I just suggest an edit: $X_{T}X$ is a positive definite matrix, not a positive semidefinite. Otherwise it would be a singular matrix, and, as such, no inverse would be available and no beta. In other words, if there is perfect multicollinearity between the regressors, then no beta exists.
– Fr1
Aug 28, 2019 at 9:19
• Thanks @Fr1, edited. Mar 11, 2022 at 8:14