# Linear Regression Coefficients

In simple linear regression, we have that given some $$(n \times 1)$$ matrix of response observations $$y$$ and a $$(n \times p)$$ matrix of observations $$x$$, the least-squares solution for $$\beta$$ is $$\beta = (x^Tx)^{-1}x^Ty.$$

I understand why this is the solution as differentiation w.r.t the MSE gives us $$x^Tx\beta - x^Ty=0$$ but doesn't this simplify to $$x^T(x\beta-y)=0$$ which can also yield the further simplified $$\beta=x^{-1}y$$.

Similarly to above, we have that $$\beta = (x^Tx)^{-1}x^Ty = x^{-1}{x^{T}}^{-1}x^Ty = x^{-1}y$$.

So if this is true why is it standard to write $$\beta$$ typically as $$\beta = (x^Tx)^{-1}x^Ty$$?

Is it just because it's assumed that $$n \neq p$$? Are there any computational benefits to the standard expression if an inverse of $$x$$ exists?

• Please explain to us what "$x^{-1}$" might possibly mean for an $n\times p$ matrix where $n\ne p.$ (Another question asked earlier today concerns the same misunderstanding: stats.stackexchange.com/questions/500247.)
– whuber
Commented Dec 10, 2020 at 21:32

You are right that because $$n \neq p$$, $$x^{-1}$$ does not exist. However, in a practical context, we wouldn't have $$x$$ be invertible. It is the case, usually, that $$n >> p$$.
The reason is if we have a square data matrix, then we have as many data points as we have regressors. Then the hyperplane defined by $$x\beta$$ will be a "perfect fit" to our data, but generally, the standard error of your estimated coefficients are going to be quite large!
Also, an important caveat you left out, to use the "least-squares" formula you describe above, you need $$x$$ to be full-rank.
If you have a full-rank, square matrix $$x$$, then, sure, you could just invert the matrix to get your least squares estimate. But, for the reasons explained above, if your data matrix is square, you probably have bigger problems on your hands.