If not minimizing SSEs, what am I doing?

Question

I've got a problem that can be summarized as a linear regression. Thus it takes the following form:

$$ Y=X \beta +\epsilon $$

where $Y$ and $\epsilon$ are vectors of size $N\times1$, $X$ is a matrix of size $N\times3$, and $\beta$ is a vector of size $3\times1$ (i.e., I am solving for three parameters).

The proper way to solve this under an ordinary least squares framework is via the normal equations—that is, taking the partial derivative of the sum of squared errors with respect to each parameter, and solving for where that equals zero in order to minimize the sum of squared errors.

However, this works rather poorly for me. On a whim, I tried just doing the following to solve for the parameters instead:

$$ \beta = X^{-1}Y $$

This actually works out really, really nicely. The problem is, I can't justify it, and I don't know what it's doing! Note that it's basically ignoring the idea that there are any errors at all.

Any ideas as to what's going on here? Thanks in advance.

Answer 1

Since the regressor matrix is not square, it does not have an inverse. I suspect though that it is of full column rank ($=3$), i.e. that your three regressors are not perfectly colinear. Then, the regressor matrix has a "left inverse", i.e. a matrix $X^{-1}_{left}$ that satisfies

$$X^{-1}_{left}X =I$$

This matrix is, ahem, equal to $(X'X)^{-1}X'$. So essentially you found

$$\beta = X^{-1}_{left}Y = (X'X)^{-1}X'Y$$ which is exactly the solution to the normal equations written in matrix notation. I guess you did this by software -if you told the software to invert the regressor matrix, it, seeing that $X$ is not a square matrix, interpreted it as asking of it to calculate the left-inverse matrix.

Execution of statistical estimation does "ignore the idea that there are errors", because these errors are unknown, and so however we want it, they cannot be included in our calculations.