9
$\begingroup$

When I have a multivariable linear regression model for sample element $i$: $$y_i=\beta_1x_{1,i}+...+\beta_kx_{k,i}+\epsilon_i,$$ then the OLS estimator is determined by the following equation: $$X^Ty=X^T X \beta.$$

Why can't we simply cancel out the $X^T$ on both sides?

And what if we coincidentally have that sample size $n=k$? Is it then possible?

$\endgroup$
5
  • 10
    $\begingroup$ What algebraic rule are you trying to invoke with "cancellation"? There is no such universal rule. Even with numbers you cannot automatically cancel equations like $xy=xb$ to conclude $y=b$: first you have to establish that $x$ has an inverse. For numbers, that means it's nonzero. For matrices, at a minimum the matrix has to be square before you can even think about inverting it. $\endgroup$
    – whuber
    Commented Oct 25, 2017 at 15:05
  • 2
    $\begingroup$ To end up with something like $y = X\beta + \varepsilon$ ..? Sure you can use something else then matrix algebra to find $\beta$ (e.g. optimization), but then you are not using OLS... $\endgroup$
    – Tim
    Commented Oct 25, 2017 at 15:06
  • 1
    $\begingroup$ @whuber The matrix does not necessarily have to be square. Sometimes you can cancel out rectangular matrices, too, with a one-sided inverse. If $X$ is a tall thin matrix with full column rank, then there exists a matrix $Y$ such that $YX=I$ (but $XY\neq I$), and you can cancel it out from equations such as $Xa=Xb$. $\endgroup$ Commented Oct 25, 2017 at 17:49
  • 2
    $\begingroup$ @Federico That's a good point about one-sided inverses. We could also proceed to discuss generalized inverses, too. But I trust we are agreed on the main point: one cannot "simply cancel out" a matrix from both sides of any equation $Xy=Xb$, for generic $y$ and $b$, without first establishing that certain special conditions apply to $X$. $\endgroup$
    – whuber
    Commented Oct 25, 2017 at 18:01
  • 2
    $\begingroup$ @FedericoPoloni, to find your matrix $Y$ you'll formulate the problem with $(XX^T)^{-1}$ in it. $\endgroup$
    – Aksakal
    Commented Oct 25, 2017 at 20:29

1 Answer 1

17
$\begingroup$

Why can't we simply cancel out the $X^T$

I remember asking myself almost exactly the same question 300 years ago upon seeing the regression equation $y=X\beta$ for the first time in my life. The difference was that I told myself: why don't we simply solve it as $\beta=X^{-1}y$? It turns out the answer is almost the same for both yours and my questions.

How does "cancelling out" work? In a simple algebra you get the following $$a\times b=a\times c$$ Then you multiply both sides by the same number, in this case it's $a^{-1}$: $$a^{-1}a\times b=a^{-1}a\times c$$ $$ b = c$$

The trouble is that neither $(X^T)^{-1}$ nor $X^{-1}$ exist when $n\ne k$. These are rectangular matrices as you can easily see, and actually allude to in your second question. There is no inversion of the rectangular matrices. I'll qualify the last statement later.

When $n=k$, you could do what I suggested in the beginning, i.e. $\beta=X^{-1}y$, because the least squares problem is not necessary anymore. It degenerates into a simple linear algebra equation with a unique solution. As noted in the comments, not even every square matrix has a solution. For instance, if you have a matrix with two identical rows there is no inverse for it.

Back to inverse of the rectangular matrix. You may have heard about the matrix pseudo inverse operation. You can apply it to invert the rectangular matrix, but there's no shortcut here: this will indeed solve the least squares equation, so you'll get back to the starting point :)

$\endgroup$
4
  • 9
    $\begingroup$ It's worth mentioning that square matrices don't necessarily have inverses either. $\endgroup$ Commented Oct 25, 2017 at 15:57
  • $\begingroup$ Where the number 307 comes frome? Sir $\endgroup$
    – Deep North
    Commented Oct 25, 2017 at 22:23
  • $\begingroup$ It's actually 300 years. Comes from a song $\endgroup$
    – Aksakal
    Commented Oct 25, 2017 at 22:27
  • 1
    $\begingroup$ @DeepNorth this is the song, the tortoise is singing "I used to be young... everything seemed amusing 300 years ago" $\endgroup$
    – Aksakal
    Commented Oct 26, 2017 at 3:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.