# How can I prove that this matrix $G$ is positive semi-definite?

$$X$$ is a $$T\times k$$ matrix of constants with $$\text{rank}(X)=k$$, $$Z$$ is a $$T\times q$$ matrix of constants with $$\text{rank}(Z)=q$$, and $$\text{rank}(X'Z)=k$$. We have that

$$P_z = Z(Z'Z)^{-1} Z'$$ and $$G= (X'P_zX)^{-1} - (X'X)^{-1}.$$

How can we prove that $$G$$ is positive semi-definite?

To prove that $$G$$ is psd is equivalent to proving that (see https://math.stackexchange.com/questions/435831/if-a-and-b-are-positive-definite-then-is-b-1-a-1-positive-semidef) $$X'X- X'P_zX$$ is psd. Write this as $$X'M_zX$$ for the residual maker matrix $$M_z=I-P_z$$.

Now, let $$d=Xc$$ for some $$c\neq0$$. Then, $$c'X'M_zXc=d'M_zd$$ By symmetry and idempotency of $$M_z$$ and letting $$e=M_zd$$, $$d'M_zd=d'M_z'M_zd=e'e=\sum_ie_i^2,$$ which is a sum of squares and hence nonnegative.

$$\DeclareMathOperator{\rank}{rank}$$

@Christoph Hanck has provided the correct solution. I just wanted to add the step of showing $$X'P_zX$$ is invertible so that it is positive definite and the result linked in Christoph Hanck's answer applies. That $$X'X$$ is invertible is relatively trivial in view of $$\operatorname{rank}(X'X) = \operatorname{rank}(X) = k$$.

To this end, we need:

Lemma: If $$A \in \mathbb{R}^{m \times n}$$ is of full column rank, then for any $$B \in \mathbb{R}^{n \times p}$$, $$\rank(AB) = \rank(B).$$

Since $$P_z$$ is symmetric and idempotent, $$X'P_zX = X'P_z'P_zX = (P_zX)'P_zX$$, it then follows that \begin{align} & \rank(X'P_zX) \\ =& \rank(P_zX) \tag*{(\rank(A'A) = \rank(A))} \\ =& \rank(Z(Z'Z)^{-1}Z'X) \tag*{(Plug in definition of P_z)}\\ =& \rank(Z'X) \tag*{(Lemma)}\\ =& \rank(X'Z) \tag*{(\rank(A) = \rank(A'))} \\ =& k. \tag*{(Condition)} \end{align} This shows that the order $$k$$ matrix $$X'P_zX$$ is invertible. In the "Lemma" step above, we also implicitly used the fact $$\rank(AB) = \rank(A)$$ if $$B$$ is invertible (where $$A = Z, B = (Z'Z)^{-1}$$).

Proof of Lemma: View $$AB$$ and $$B$$ as linear mappings from $$\mathbb{R}^{p}$$ to $$\mathbb{R}^m$$ and $$\mathbb{R}^p$$ to $$\mathbb{R}^n$$ respectively, then by rank-nullity theorem, $$\rank(AB) = p - \dim(N(AB))$$, $$\rank(B) = p - \dim(N(B))$$. Therefore it suffices to show $$N(AB) = N(B)$$. Clearly $$N(B) \subset N(AB)$$. Conversely, if $$x \in N(AB)$$, then $$ABx = 0$$, whence $$Bx \in N(A) = 0$$ due to $$\rank(A) = n$$ (apply rank-nullity theorem again). That means $$x \in N(B)$$, i.e., $$N(AB) = N(B)$$. This completes the proof.