ELI5: The Logic Behind Coefficient Estimation in OLS Regression

Question

Like a lot of people, I understand how to run a linear regression, I understand how to interpret its output, and I understand its limitations.

My understanding of the mathematical underpinnings of linear regression, however, are less developed. In particular, I do not understand the logic behind how we estimate beta using the following formula:

$$ \beta = (X'X)^{-1}X'Y $$

Would anyone care to offer an intuitive explanation as to why/how this process works? For example, what function each step in the equation performs and why it is necessary.

Answer 1

Suppose you have a model of the form:
$$X \beta= Y$$ where X is a normal 2-D matrix, for ease of visualisation. Now, if the matrix $X$ is square and invertible, then getting $\beta$ is trivial: $$\beta= X^{-1}Y$$ And that would be the end of it.

If this is not the case, to get $\beta$ you’ll have to find a way to “approximate” the result of an inverse matrix. $X^\dagger = (X'X)^{-1}X'$ is called the (left)-pseudoinverse, and it has some nice properties that make it useful for this application.

In particular, it is unique, and $XX^\dagger X=X$, so it kind of works like an inverse matrix would $(XX^{-1}X = XI = X)$. Also, for an invertible and square matrix (i.e. if the inverse matrix exists), it is equal to $X^{-1}$.

Also it gets the shape of the matrix right: If $X$ has order $n \times m$, our pseudoinverse should be $m \times n$ so we can multiply it with $Y$. This is achieved by multiplying $(X'X)^{-1}$, which is square $(m \times m)$, with X' $(m \times n)$.

Answer 2

If you look at sources such as wikipedia, there are some good explanations for where this comes from. Here are some cores ideas:

1. OLS is aiming to minimize the error $||y-X\beta||$.

2. The norm of a vector is minimized when its derivative is perpendicular to the vector. (Since you asked for ELI5, I won't go into a rigorous formulation of "derivative" in this context.)

3. The error is given in terms of $y$, $X$, and $\beta$. The first two are constants; we're varying only $\beta$. Thus, the derivative can be treated as being $X\beta'$, so we're looking for $(X\beta')^T(y-X\beta)=0$. This is equivalent to $(\beta')^TX^Ty=(\beta')^TX^TX\beta$. If we cancel the $(\beta')^T$ from both sides (normally in linear algebra, you can't just go around canceling things, but I'm not aiming for perfect rigor here, so I won't get into the justification), we're left with $X^Ty=X^TX\beta$. Now, $X^T$ isn't invertible (it isn't even square), so we can't cancel it out, but it does turn out that $X^TX$ must be invertible (assuming that the features are linearly independent). So we can get $\beta = (X^TX)^{-1}X^Ty$.

Going back to $X^Ty=X^TX\beta$, recall that $X\beta$ is the the estimate $\hat y$ that is calculated from a given $\beta$. $X^Ty$ is a vector in which each entry is the dot product of one of the features with the response. So we have that $X^Ty=X^T\hat y$, i.e., for each feature, the dot product between that feature and the actual response is equal to the dot product between that feature and the estimated response. $\forall i, x_i^Ty=x_i^T\hat y$. We can view OLS, then, as solving $n$ equations $x_i^Ty=x_i^T\hat y$, where $n$ is the number of features. So to see why this works, we just need to show that a solution exists, and that any estimate of the response other than this solution will have larger squared error.