Like a lot of people, I understand how to run a linear regression, I understand how to interpret its output, and I understand its limitations.

My understanding of the mathematical underpinnings of linear regression, however, are less developed. In particular, I do not understand the logic behind how we estimate beta using the following formula:

$$ \beta = (X'X)^{-1}X'Y $$

Would anyone care to offer an intuitive explanation as to why/how this process works? For example, what function each step in the equation performs and why it is necessary.

  • 7
    $\begingroup$ How many five year olds have learned anything about algebra, let alone matrices? I don't think it's a feasible request. Better to be clear about what kind/level of explanation you realistically seek. It would also help to clarify what it is you seek (that's not especially clear); are you asking for some outline explanation of how the formula is derived, or why a formula something like that makes sense? $\endgroup$ – Glen_b Dec 11 '18 at 11:52

Suppose you have a model of the form:
$$X \beta= Y$$ where X is a normal 2-D matrix, for ease of visualisation. Now, if the matrix $X$ is square and invertible, then getting $\beta$ is trivial: $$\beta= X^{-1}Y$$ And that would be the end of it.

If this is not the case, to get $\beta$ you’ll have to find a way to “approximate” the result of an inverse matrix. $X^\dagger = (X'X)^{-1}X'$ is called the (left)-pseudoinverse, and it has some nice properties that make it useful for this application.

In particular, it is unique, and $XX^\dagger X=X$, so it kind of works like an inverse matrix would $(XX^{-1}X = XI = X)$. Also, for an invertible and square matrix (i.e. if the inverse matrix exists), it is equal to $X^{-1}$.

Also it gets the shape of the matrix right: If $X$ has order $n \times m$, our pseudoinverse should be $m \times n$ so we can multiply it with $Y$. This is achieved by multiplying $(X'X)^{-1}$, which is square $(m \times m)$, with X' $(m \times n)$.

  • $\begingroup$ Thanks for your time. This was a great explanation and really useful. $\endgroup$ – Jack Bailey Dec 11 '18 at 14:40

If you look at sources such as wikipedia, there are some good explanations for where this comes from. Here are some cores ideas:

  1. OLS is aiming to minimize the error $||y-X\beta||$.

  2. The norm of a vector is minimized when its derivative is perpendicular to the vector. (Since you asked for ELI5, I won't go into a rigorous formulation of "derivative" in this context.)

  3. The error is given in terms of $y$, $X$, and $\beta$. The first two are constants; we're varying only $\beta$. Thus, the derivative can be treated as being $X\beta'$, so we're looking for $(X\beta')^T(y-X\beta)=0$. This is equivalent to $(\beta')^TX^Ty=(\beta')^TX^TX\beta$. If we cancel the $(\beta')^T$ from both sides (normally in linear algebra, you can't just go around canceling things, but I'm not aiming for perfect rigor here, so I won't get into the justification), we're left with $X^Ty=X^TX\beta$. Now, $X^T$ isn't invertible (it isn't even square), so we can't cancel it out, but it does turn out that $X^TX$ must be invertible (assuming that the features are linearly independent). So we can get $\beta = (X^TX)^{-1}X^Ty$.

Going back to $X^Ty=X^TX\beta$, recall that $X\beta$ is the the estimate $\hat y$ that is calculated from a given $\beta$. $X^Ty$ is a vector in which each entry is the dot product of one of the features with the response. So we have that $X^Ty=X^T\hat y$, i.e., for each feature, the dot product between that feature and the actual response is equal to the dot product between that feature and the estimated response. $\forall i, x_i^Ty=x_i^T\hat y$. We can view OLS, then, as solving $n$ equations $x_i^Ty=x_i^T\hat y$, where $n$ is the number of features. So to see why this works, we just need to show that a solution exists, and that any estimate of the response other than this solution will have larger squared error.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.