ELI5: The Logic Behind Coefficient Estimation in OLS Regression

Like a lot of people, I understand how to run a linear regression, I understand how to interpret its output, and I understand its limitations.

My understanding of the mathematical underpinnings of linear regression, however, are less developed. In particular, I do not understand the logic behind how we estimate beta using the following formula:

$$\beta = (X'X)^{-1}X'Y$$

Would anyone care to offer an intuitive explanation as to why/how this process works? For example, what function each step in the equation performs and why it is necessary.

• How many five year olds have learned anything about algebra, let alone matrices? I don't think it's a feasible request. Better to be clear about what kind/level of explanation you realistically seek. It would also help to clarify what it is you seek (that's not especially clear); are you asking for some outline explanation of how the formula is derived, or why a formula something like that makes sense? – Glen_b Dec 11 '18 at 11:52

Suppose you have a model of the form:
$$X \beta= Y$$ where X is a normal 2-D matrix, for ease of visualisation. Now, if the matrix $$X$$ is square and invertible, then getting $$\beta$$ is trivial: $$\beta= X^{-1}Y$$ And that would be the end of it.

If this is not the case, to get $$\beta$$ you’ll have to find a way to “approximate” the result of an inverse matrix. $$X^\dagger = (X'X)^{-1}X'$$ is called the (left)-pseudoinverse, and it has some nice properties that make it useful for this application.

In particular, it is unique, and $$XX^\dagger X=X$$, so it kind of works like an inverse matrix would $$(XX^{-1}X = XI = X)$$. Also, for an invertible and square matrix (i.e. if the inverse matrix exists), it is equal to $$X^{-1}$$.

Also it gets the shape of the matrix right: If $$X$$ has order $$n \times m$$, our pseudoinverse should be $$m \times n$$ so we can multiply it with $$Y$$. This is achieved by multiplying $$(X'X)^{-1}$$, which is square $$(m \times m)$$, with X' $$(m \times n)$$.

• Thanks for your time. This was a great explanation and really useful. – Jack Bailey Dec 11 '18 at 14:40

If you look at sources such as wikipedia, there are some good explanations for where this comes from. Here are some cores ideas:

1. OLS is aiming to minimize the error $$||y-X\beta||$$.

2. The norm of a vector is minimized when its derivative is perpendicular to the vector. (Since you asked for ELI5, I won't go into a rigorous formulation of "derivative" in this context.)

3. The error is given in terms of $$y$$, $$X$$, and $$\beta$$. The first two are constants; we're varying only $$\beta$$. Thus, the derivative can be treated as being $$X\beta'$$, so we're looking for $$(X\beta')^T(y-X\beta)=0$$. This is equivalent to $$(\beta')^TX^Ty=(\beta')^TX^TX\beta$$. If we cancel the $$(\beta')^T$$ from both sides (normally in linear algebra, you can't just go around canceling things, but I'm not aiming for perfect rigor here, so I won't get into the justification), we're left with $$X^Ty=X^TX\beta$$. Now, $$X^T$$ isn't invertible (it isn't even square), so we can't cancel it out, but it does turn out that $$X^TX$$ must be invertible (assuming that the features are linearly independent). So we can get $$\beta = (X^TX)^{-1}X^Ty$$.

Going back to $$X^Ty=X^TX\beta$$, recall that $$X\beta$$ is the the estimate $$\hat y$$ that is calculated from a given $$\beta$$. $$X^Ty$$ is a vector in which each entry is the dot product of one of the features with the response. So we have that $$X^Ty=X^T\hat y$$, i.e., for each feature, the dot product between that feature and the actual response is equal to the dot product between that feature and the estimated response. $$\forall i, x_i^Ty=x_i^T\hat y$$. We can view OLS, then, as solving $$n$$ equations $$x_i^Ty=x_i^T\hat y$$, where $$n$$ is the number of features. So to see why this works, we just need to show that a solution exists, and that any estimate of the response other than this solution will have larger squared error.