# Showing that the difference of two variance matrices should be positive semi-definite… or not

I am trying to prove a result: intuitively, the GMM estimator should have a larger variance under homoskedasticity than LS and vice versa under heteroskedasticity. We are working with various probability (VP) sampling with $b(X_i)$ being the probability of $i$th observation to be retained in the sample, so, in order to account for stratification, we do inverse probability weighting:

$\mathbb{E}\frac{X(Y-X'\theta)}{b(X)} = 0$

The asymptotic variances of the two estimators are the following:

$\mathrm{asVar}(LS) = (\mathbb{E}XX')^{-1}(\mathbb{E}XX'\sigma^2(X))(\mathbb{E}XX')^{-1}$ $\mathrm{asVar}(GMM) = \left(\mathbb{E}\frac{XX'}{b(X)}\right)^{-1} \left(\mathbb{E}\frac{XX'\sigma^2(X)}{b^2(X)}\right) \left(\mathbb{E}\frac{XX'}{b(X)}\right)^{-1}$

I have run multiple simulations with various parameters, and since we are working in heteroskedastic conditions, it seems that the variance of the GMM estimator should beat that of the LS one, but (1) in the simulations, the variance of GMM estimator remained larger even under heteroskedasticity, and (2) so far I could not prove that any ranking should exist. This is the idea of the proof.

Let $\Omega = \mathbb{E}XX'\frac{\sigma^2(X)}{b^2(X)} - \left(\mathbb{E}\frac{XX'}{b(X)}\right) (\mathbb{E}XX')^{-1}(\mathbb{E}XX'\sigma^2(X))(\mathbb{E}XX')^{-1} \left(\mathbb{E}\frac{XX'}{b(X)}\right)$. Then

$\mathrm{asVar}(GMM) - \mathrm{asVar}(LS) = \left(\mathbb{E}\frac{XX'}{b(X)}\right)^{-1} \Omega \left(\mathbb{E}\frac{XX'}{b(X)}\right)^{-1},$ so we focus our attention on $\Omega$. It can be decomposed into a sum of two matrices.

Let $a_1 = X \frac{\sigma(X)}{b(X)}$ and $a_2 = (EXX')^{-1} \frac{X}{\sigma(X)}$. Then $\mathbb{E} a_1 a_1' = \mathbb{E} XX' \frac{\sigma^2(X)}{b^2(X)}$, $\mathbb{E} a_1 a_2' = \mathbb{E} \frac{XX'}{b(X)} (\mathbb{E}XX')^{-1}$ and $\mathbb{E} a_2 a_2' = (\mathbb{E}XX')^{-1} \mathbb{E}\frac{XX'}{\sigma^2(X)} (\mathbb{E}XX')^{-1}$.

Now we can simplify $\Omega$: $\Omega = \mathbb{E} a_1 a_1' - \mathbb{E} a_1 a_2' \ [\mathbb{E}XX' \sigma^2(X) ]\ \mathbb{E} a_2 a_1'$

Let us subtract and add $(\mathbb{E} a_2 a_2')^{-1}$ to the term in brackets:

$\Omega = \mathbb{E} a_1 a_1' - \mathbb{E} a_1 a_2' \ [\mathbb{E}XX' \sigma^2(X) - (\mathbb{E} a_2 a_2')^{-1} + (\mathbb{E} a_2 a_2')^{-1}]\ \mathbb{E} a_2 a_1'$

And now we expand:

$\Omega = \underbrace{\mathbb{E} a_1 a_1' - \mathbb{E} a_1 a_2' \ (\mathbb{E} a_2 a_2')^{-1} \ \mathbb{E} a_2 a_1'}_A + \underbrace{\mathbb{E} a_1 a_2' \ [(\mathbb{E} a_2 a_2')^{-1} - \mathbb{E}XX' \sigma^2(X)] \ \mathbb{E} a_2 a_1'}_B$

So basically $\Omega = A + B$. We can show that $A$ is positive semi-definite.

Matrix analogue of Cauchy-Schwarz inequality. If $g$ and $h$ are random vectors of the same length, then $(\mathbb{E} gh') (\mathbb{E}hh')^{-1}(\mathbb{E} hg') \leq \mathbb{E} gg'$ in the sense that their difference is a positive semi-definite matrix.

Using the matrix analogue of the Cauchy-Schwarz inequality, we can see that $\mathbb{E} a_1 a_1' - \mathbb{E} a_1 a_2' \ (\mathbb{E} a_2 a_2')^{-1} \ \mathbb{E} a_2 a_1'$ is positive semi-definite.

Now for $B$. Again, we are interested in the “meat” of the sandwich, namely $(\mathbb{E} a_2 a_2')^{-1} - \mathbb{E}XX' \sigma^2(X)$. In this case, by the very same inequality, this matrix is negative semi-definite: consider $g=X\sigma(X)$, $h=\frac{X}{\sigma(X)}$, so

$(\mathbb{E} gh') (\mathbb{E}hh')^{-1}(\mathbb{E} hg') = (\mathbb{E} XX') \left(\mathbb{E} \frac{XX'}{\sigma^2(X)}\right)^{-1} (\mathbb{E} XX') \le \mathbb{E}XX' \sigma^2(X) = \mathbb{E} gg'.$

In case of homoskedasticity, $\sigma^2(X) \equiv \sigma^2$, to the latter inequality turns into equality because then, $(\mathbb{E} XX') \left(\mathbb{E} \frac{XX'}{\sigma^2}\right)^{-1} (\mathbb{E} XX') \equiv \mathbb{E}XX' \sigma^2$, so $\Omega=A$ under homoskedasticity. However, it seems logical that at some point in time, under, $B$ should outweigh $A$ and make $\Omega$ NSD... or not?

And there we have it. $\Omega$ turns out to be a sum of a PSD and a NSD matrix. So adding and subtracting $(\mathbb{E}a_2 a_2')^{-1}$ was a convenient but a useless trick.

Question. Is it possible to show that $\Omega$ is positive semi-definite at all? Or is there a counterexample showing the contrary?

Indeed, no such ranking exists. I ran another simulation in a very simplified case (two-point distribution of $X$, two values of $\sigma^2(X)$, two different probabilities of being retained) and, depending on the probabilities, the inequality sign flipped.