2
$\begingroup$

I understand the derivation process of the least square estimator of B 𝑏=(𝑋′𝑋)-1 𝑋′𝑦

However, I am not seeing what I am doing wrong here, because I feel this can be further simplified like

𝑏 = (𝑋′𝑋)-1 𝑋′𝑦
= (𝑋)-1 (𝑋′)-1 𝑋′𝑦
= (𝑋)-1 Y

where am I doing wrong, please point out. Thank You!

Ok! I see , it has to be sqaure matrix to clearly define the inverse, I am just confused with derivation of the following:enter image description here

Not sure how it gets from first red line to the second redline.. Thank you! Any help be will appreacited!

$\endgroup$
1
  • 1
    $\begingroup$ You are assuming your equations make sense. Could you explain what, say, "$X^{-1}$" means when $X$ is not a square matrix?? $\endgroup$
    – whuber
    Commented Oct 23, 2019 at 19:35

1 Answer 1

2
$\begingroup$

If $X$ and therefore $X^T$ are indeed square and invertible then you are correct: the linear system $y = X\beta$ can be solved by inversion to get $\beta = X^{-1}y$ (and the residuals are all zero). But in linear regression we usually have $n \gg p$ so rather than exactly solving that system, we instead find the element of the column space of $X$ that is as close as possible to $y$, which likely is not in that $p$-dimensional subspace. This element is $\hat y = X\hat\beta$, and $\hat\beta$ is found via $\hat\beta = X^{+}y$ where $X^+$ is the pseudoinverse rather than the actual inverse.

The equation $(X^TX)^{-1} \stackrel ?= X^{-1}X^{-T}$ does not hold when $X$ is not square because $X$ is not invertible so that is not meaningful in general (as @whuber says in his comment).


For your update, we are using the fact that $$ (ABC)^{-1}=C^{-1}B^{-1}A^{-1} $$ for invertible $A$, $B$, and $C$, but we're still never pulling $X^TX$ apart, we're just doing $$ ((XD)^T(XD))^{-1}(XD)^T = \\ (DX^TXD)^{-1}DX \\ = D^{-1}(X^TX)^{-1}D^{-1}DX^T \\ = D^{-1}(X^TX)^{-1}X^T $$ so once we start using inverses $X^TX$ is only ever considered as a single square matrix, the fact that it's coming from $X$ isn't relevant as far as the inversion is concerned.

$\endgroup$
3
  • $\begingroup$ Yes! I just caught that! I have edited some examples that I am really confused about, where X is not necessarily a square matrix. Thank you so much! $\endgroup$ Commented Oct 23, 2019 at 19:49
  • $\begingroup$ @WendyHuang ok, no prob, I just updated $\endgroup$
    – jld
    Commented Oct 23, 2019 at 19:51
  • $\begingroup$ very clear thank you@ild $\endgroup$ Commented Oct 23, 2019 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.