Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
$X$ is a $T\times k$ matrix of constants with $\text{rank}(X)=k$, $Z$ is a $T\times q$ matrix of constants with $\text{rank}(Z)=q$, and $\text{rank}(X'Z)=k$. We have that
$$P_z = Z(Z'Z)^{-1} Z'$$ and $$G= (X'P_zX)^{-1} - (X'X)^{-1}.$$
How can we prove that $G$ is positive semi-definite?
To prove that $G$ is psd is equivalent to proving that (see https://math.stackexchange.com/questions/435831/if-a-and-b-are-positive-definite-then-is-b-1-a-1-positive-semidef) $$ X'X- X'P_zX $$ is psd. Write this as $X'M_zX$ for the residual maker matrix $M_z=I-P_z$.
Now, let $d=Xc$ for some $c\neq0$. Then, $$ c'X'M_zXc=d'M_zd $$ By symmetry and idempotency of $M_z$ and letting $e=M_zd$, $$ d'M_zd=d'M_z'M_zd=e'e=\sum_ie_i^2, $$ which is a sum of squares and hence nonnegative.
$\DeclareMathOperator{\rank}{rank}$
@Christoph Hanck has provided the correct solution. I just wanted to add the step of showing $X'P_zX$ is invertible so that it is positive definite and the result linked in Christoph Hanck's answer applies. That $X'X$ is invertible is relatively trivial in view of $\operatorname{rank}(X'X) = \operatorname{rank}(X) = k$.
To this end, we need:
Lemma: If $A \in \mathbb{R}^{m \times n}$ is of full column rank, then for any $B \in \mathbb{R}^{n \times p}$, $$\rank(AB) = \rank(B).$$
Since $P_z$ is symmetric and idempotent, $X'P_zX = X'P_z'P_zX = (P_zX)'P_zX$, it then follows that \begin{align} & \rank(X'P_zX) \\ =& \rank(P_zX) \tag*{($\rank(A'A) = \rank(A)$)} \\ =& \rank(Z(Z'Z)^{-1}Z'X) \tag*{(Plug in definition of $P_z$)}\\ =& \rank(Z'X) \tag*{(Lemma)}\\ =& \rank(X'Z) \tag*{($\rank(A) = \rank(A')$)} \\ =& k. \tag*{(Condition)} \end{align} This shows that the order $k$ matrix $X'P_zX$ is invertible. In the "Lemma" step above, we also implicitly used the fact $\rank(AB) = \rank(A)$ if $B$ is invertible (where $A = Z, B = (Z'Z)^{-1}$).
Proof of Lemma: View $AB$ and $B$ as linear mappings from $\mathbb{R}^{p}$ to $\mathbb{R}^m$ and $\mathbb{R}^p$ to $\mathbb{R}^n$ respectively, then by rank-nullity theorem, $\rank(AB) = p - \dim(N(AB))$, $\rank(B) = p - \dim(N(B))$. Therefore it suffices to show $N(AB) = N(B)$. Clearly $N(B) \subset N(AB)$. Conversely, if $x \in N(AB)$, then $ABx = 0$, whence $Bx \in N(A) = 0$ due to $\rank(A) = n$ (apply rank-nullity theorem again). That means $x \in N(B)$, i.e., $N(AB) = N(B)$. This completes the proof.
