How to prove positive (semi-)definitness in matrix notation without numbers

Question

I'd like to show that $$V[\hat\beta_{OLS}]-V[\hat\beta_{GLS}]=\sigma^2(X'X)^{-1}(X'\Omega X)(X'X)^{-1}-\sigma^2(X'\Omega X)^{-1}\geq 0$$ is positive (semi-) definite. $\Omega$ and $X$ are square-matrices. I know how to check for positive (semi-) definiteness with eigenvalues if the matrices contain numbers, but I wonder if there is a quick way to check if the above expression is positive (semi-) definite when we only have matrix notation without numbers?

Answer 1

Note that $X$ is generally not square, unless you have as many regressors as observations. In that case - see the derivation at the end of my answer - OLS and GLS are equally efficient.

Notice also a mistake in the above expression in that the correct variance of GLS is $$ \sigma^2(X'\Omega^{-1} X)^{-1} $$

A standard result in matrix algebra tells out that $$ A\geq B\Leftrightarrow B^{-1}-A^{-1}\geq0$$ (much like $3>2$ and $1/2>1/3$). Hence, we may also establish that ($\sigma^2$ is just a scale factor which cancels out of the comparison) $$ X'\Omega^{-1} X-X'X(X'\Omega X)^{-1}X'X\geq 0 $$ Rearrange the lhs to $$ X'(\Omega^{-1}-X(X'\Omega X)^{-1}X')X $$ or, with $\Omega^{1/2}$ a symmetric matrix square root of $\Omega$ and $\Omega^{-1/2}$ its inverse, $$ X'(\Omega^{-1/2}\Omega^{-1/2}-\Omega^{-1/2}\Omega^{1/2}X(X'\Omega X)^{-1}X'\Omega^{1/2}\Omega^{-1/2})X $$ or $$ X'\Omega^{-1/2}(I-\Omega^{1/2}X(X'\Omega X)^{-1}X'\Omega^{1/2})\Omega^{-1/2}X $$ or $$ X'\Omega^{-1/2}(I-\Omega^{1/2}X(X'\Omega^{1/2}\Omega^{1/2} X)^{-1}X'\Omega^{1/2})\Omega^{-1/2}X $$ Now let $C:=\Omega^{-1/2}X$ and $D:=\Omega^{1/2}X$ so that we may write $$ C'(I-D(D'D)^{-1}D')C $$ Now, $M_D:=I-D(D'D)^{-1}D'$ is a residual-maker, hence, symmetric and idempotent projection matrix. Let $c:=Ce$ for some nonzero vector $e$. Then, for $f:=M_Dc$, $$ c'M_Dc=c'M_D'M_Dc\equiv f'f\geq0, $$ as $f'f=\sum_if_i^2$ is a sum of squares.

Now, if $X$ actually were square (and invertible), we would have $$ X'X(X'\Omega X)^{-1}X'X=X'XX^{-1}\Omega^{-1}(X')^{-1}X'X=X'\Omega^{-1} X $$ so that OLS and GLS would be equally efficient!

In fact, they would be numerically identical, as $$ (X'\Omega^{-1} X)^{-1}X'\Omega^{-1}y=X^{-1}\Omega X'^{-1}X'\Omega^{-1}y=X^{-1}y=X^{-1}X'^{-1}X'y=(X'X)^{-1}X'y $$