I would like to know why $ \beta = (X^T X)^{-1} X^T y $ is the solution for the ML-estimator for the linear regression.

In principle it is shown, e.g., here in this file respectively in specific I would like to know where the inverse comes from?

  • 1
    $\begingroup$ The answer is given here although the question is different. Essentially, you just have to show that $\hat{\beta}$ achieves your objective (either minimizing square error or maximizing likelihood, which are equivalent in this case). stats.stackexchange.com/questions/286497/… $\endgroup$
    – Sycorax
    Aug 28, 2019 at 12:55

1 Answer 1


in the notes you use, $X$ is a $T \times k$ matrix, to find the extrema one needs to set the right-hand side of Eq. 7 to zero, which is equivalent to solving

$(y - X \beta)' = 0 $

as it should vanish for every $X$. Note on the previous equation $y$, and $\beta$ are vectors while $X$ is a matrix, the zero is actually a $T$-dimensional vector of zeroes.

This eq is a linear system, but the matrix $X$ is not invertible in general, so you cannot solve it as $\beta = X^{-1} y$ since $ X^{-1}$ would not be defined. You can do a trick to bypass this, multiply each side by $X^T$ (the transpose) and then you have

$( X^T y - (X^T X ) \beta)' = 0 $

Now $X^T X$ is a positive definite symmetric matrix so we know it is invertible, now you can use the usual solution for a linear system

$\beta = (X^T X )^{-1} X^T y $

which is what you have.

  • 4
    $\begingroup$ Good answer, I just suggest an edit: $X_{T}X$ is a positive definite matrix, not a positive semidefinite. Otherwise it would be a singular matrix, and, as such, no inverse would be available and no beta. In other words, if there is perfect multicollinearity between the regressors, then no beta exists. $\endgroup$
    – Fr1
    Aug 28, 2019 at 9:19
  • $\begingroup$ Thanks @Fr1, edited. $\endgroup$ Mar 11, 2022 at 8:14

Not the answer you're looking for? Browse other questions tagged or ask your own question.