6
$\begingroup$

When I have a multivariable linear regression model for sample element $i$: $$y_i=\beta_1x_{1,i}+...+\beta_kx_{k,i}+\epsilon_i,$$ then the OLS estimator is determined by the following equation: $$X^Ty=X^T X \beta.$$

Why can't we simply cancel out the $X^T$ on both sides?

And what if we coincidentally have that sample size $n=k$? Is it then possible?

$\endgroup$
  • 9
    $\begingroup$ What algebraic rule are you trying to invoke with "cancellation"? There is no such universal rule. Even with numbers you cannot automatically cancel equations like $xy=xb$ to conclude $y=b$: first you have to establish that $x$ has an inverse. For numbers, that means it's nonzero. For matrices, at a minimum the matrix has to be square before you can even think about inverting it. $\endgroup$ – whuber Oct 25 '17 at 15:05
  • 2
    $\begingroup$ To end up with something like $y = X\beta + \varepsilon$ ..? Sure you can use something else then matrix algebra to find $\beta$ (e.g. optimization), but then you are not using OLS... $\endgroup$ – Tim Oct 25 '17 at 15:06
  • 1
    $\begingroup$ @whuber The matrix does not necessarily have to be square. Sometimes you can cancel out rectangular matrices, too, with a one-sided inverse. If $X$ is a tall thin matrix with full column rank, then there exists a matrix $Y$ such that $YX=I$ (but $XY\neq I$), and you can cancel it out from equations such as $Xa=Xb$. $\endgroup$ – Federico Poloni Oct 25 '17 at 17:49
  • 2
    $\begingroup$ @Federico That's a good point about one-sided inverses. We could also proceed to discuss generalized inverses, too. But I trust we are agreed on the main point: one cannot "simply cancel out" a matrix from both sides of any equation $Xy=Xb$, for generic $y$ and $b$, without first establishing that certain special conditions apply to $X$. $\endgroup$ – whuber Oct 25 '17 at 18:01
  • 2
    $\begingroup$ @FedericoPoloni, to find your matrix $Y$ you'll formulate the problem with $(XX^T)^{-1}$ in it. $\endgroup$ – Aksakal Oct 25 '17 at 20:29
13
$\begingroup$

Why can't we simply cancel out the $X^T$

I remember asking myself almost exactly the same question 300 years ago upon seeing the regression equation $y=X\beta$ for the first time in my life. The difference was that I told myself: why don't we simply solve it as $\beta=X^{-1}y$? It turns out the answer is almost the same for both yours and my questions.

How does "cancelling out" work? In a simple algebra you get the following $$a\times b=a\times c$$ Then you multiply both sides by the same number, in this case it's $a^{-1}$: $$a^{-1}a\times b=a^{-1}a\times c$$ $$ b = c$$

The trouble is that neither $(X^T)^{-1}$ nor $X^{-1}$ exist when $n\ne k$. These are rectangular matrices as you can easily see, and actually allude to in your second question. There is no inversion of the rectangular matrices. I'll qualify the last statement later.

When $n=k$, you could do what I suggested in the beginning, i.e. $\beta=X^{-1}y$, because the least squares problem is not necessary anymore. It degenerates into a simple linear algebra equation with a unique solution. As noted in the comments, not even every square matrix has a solution. For instance, if you have a matrix with two identical rows there is no inverse for it.

Back to inverse of the rectangular matrix. You may have heard about the matrix pseudo inverse operation. You can apply it to invert the rectangular matrix, but there's no shortcut here: this will indeed solve the least squares equation, so you'll get back to the starting point :)

$\endgroup$
  • 9
    $\begingroup$ It's worth mentioning that square matrices don't necessarily have inverses either. $\endgroup$ – Matthew Drury Oct 25 '17 at 15:57
  • $\begingroup$ Where the number 307 comes frome? Sir $\endgroup$ – Deep North Oct 25 '17 at 22:23
  • $\begingroup$ It's actually 300 years. Comes from a song $\endgroup$ – Aksakal Oct 25 '17 at 22:27
  • 1
    $\begingroup$ @DeepNorth this is the song, the tortoise is singing "I used to be young... everything seemed amusing 300 years ago" $\endgroup$ – Aksakal Oct 26 '17 at 3:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.