$X$ is a $T\times k$ matrix of constants with $\text{rank}(X)=k$, $Z$ is a $T\times q$ matrix of constants with $\text{rank}(Z)=q$, and $\text{rank}(X'Z)=k$. We have that

$$P_z = Z(Z'Z)^{-1} Z'$$ and $$G= (X'P_zX)^{-1} - (X'X)^{-1}.$$

How can we prove that $G$ is positive semi-definite?


2 Answers 2


To prove that $G$ is psd is equivalent to proving that (see https://math.stackexchange.com/questions/435831/if-a-and-b-are-positive-definite-then-is-b-1-a-1-positive-semidef) $$ X'X- X'P_zX $$ is psd. Write this as $X'M_zX$ for the residual maker matrix $M_z=I-P_z$.

Now, let $d=Xc$ for some $c\neq0$. Then, $$ c'X'M_zXc=d'M_zd $$ By symmetry and idempotency of $M_z$ and letting $e=M_zd$, $$ d'M_zd=d'M_z'M_zd=e'e=\sum_ie_i^2, $$ which is a sum of squares and hence nonnegative.



@Christoph Hanck has provided the correct solution. I just wanted to add the step of showing $X'P_zX$ is invertible so that it is positive definite and the result linked in Christoph Hanck's answer applies. That $X'X$ is invertible is relatively trivial in view of $\operatorname{rank}(X'X) = \operatorname{rank}(X) = k$.

To this end, we need:

Lemma: If $A \in \mathbb{R}^{m \times n}$ is of full column rank, then for any $B \in \mathbb{R}^{n \times p}$, $$\rank(AB) = \rank(B).$$

Since $P_z$ is symmetric and idempotent, $X'P_zX = X'P_z'P_zX = (P_zX)'P_zX$, it then follows that \begin{align} & \rank(X'P_zX) \\ =& \rank(P_zX) \tag*{($\rank(A'A) = \rank(A)$)} \\ =& \rank(Z(Z'Z)^{-1}Z'X) \tag*{(Plug in definition of $P_z$)}\\ =& \rank(Z'X) \tag*{(Lemma)}\\ =& \rank(X'Z) \tag*{($\rank(A) = \rank(A')$)} \\ =& k. \tag*{(Condition)} \end{align} This shows that the order $k$ matrix $X'P_zX$ is invertible. In the "Lemma" step above, we also implicitly used the fact $\rank(AB) = \rank(A)$ if $B$ is invertible (where $A = Z, B = (Z'Z)^{-1}$).

Proof of Lemma: View $AB$ and $B$ as linear mappings from $\mathbb{R}^{p}$ to $\mathbb{R}^m$ and $\mathbb{R}^p$ to $\mathbb{R}^n$ respectively, then by rank-nullity theorem, $\rank(AB) = p - \dim(N(AB))$, $\rank(B) = p - \dim(N(B))$. Therefore it suffices to show $N(AB) = N(B)$. Clearly $N(B) \subset N(AB)$. Conversely, if $x \in N(AB)$, then $ABx = 0$, whence $Bx \in N(A) = 0$ due to $\rank(A) = n$ (apply rank-nullity theorem again). That means $x \in N(B)$, i.e., $N(AB) = N(B)$. This completes the proof.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.