Multivariate Cramér-Rao inequality: intuition for positive semidefiniteness

Question

Here's what Wikipedia says about the Multivariate Cramér-Rao inequality:

If $\boldsymbol{T}(X)$ is an unbiased estimator of $\boldsymbol{\theta}$, then the Cramér–Rao bound reduces to $\mathrm{cov}_{\boldsymbol{\theta}}\left(\boldsymbol{T}(X)\right) \geq I\left(\boldsymbol{\theta}\right)^{-1}$.

The matrix inequality $A \ge B$ is understood to mean that the matrix $A-B$ is positive semidefinite.

I understand everything above. But I started playing around with examples and came up with something that doesn't make sense to me. Suppose we have two unbiased estimators of some two-dimensional $\boldsymbol{\theta}$, one with covariance matrix

$A=\left[\begin{array}{cc} 3 & 1.5\\ 1.5 & 3 \end{array}\right]$

and the other with covariance matrix

$B=\left[\begin{array}{cc} 2 & 0.3\\ 0.3 & 2 \end{array}\right]$.

Now $A-B$ is not positive semidefinite. So even though each parameter estimate has a lower variance in $B$, and there's less covariance between the two of them, $B$ doesn't 'count' as having lower variance than A in the psd sense.

Can someone give an intuitive explanation why? (I guess maybe I'm really asking why is the generalized inequality with respect to the psd cone the 'right' comparison here? Or is it just the only one for which we can prove this result?)

Answer 1

I find it useful to think of the Convolution Theorem: a large-sample independent-observation version.

It says that (asymptotically), for any regular estimator $T$ we have $$T(X)-\theta= Z+ \Delta$$ where $Z$ has covariance matrix $I^{-1}(\theta)$ and $\Delta$ is independent of $Z$. That is, any estimator is the efficient estimator plus pure noise. Which is a nice property for an efficiency bound to have. [Another version says this holds for any estimator $T(X)$ at almost all $\theta$]

If you look at what this says about $\mathrm{cov}[T(X)]$, it is exactly that $$\mathrm{cov}[T(X)]\geq I^{-1}(\theta)$$

Things are more complicated in small samples-- for example, there need not be a fully efficient estimator -- but the large-sample motivation is still relevant.

Answer 2

So the covariance matrices are $A=(\begin{smallmatrix} 3 & 1.5 \\ 1.5 & 3\end{smallmatrix})$ and $B=(\begin{smallmatrix} 2 & 0.3 \\ 0.3 2\end{smallmatrix})$. Yes, visually it seems like $A$ is "bigger than $B$, and really, that holds for the determinants "generalized variances" which are 6.75 and 3.91. But that impression is deceiving. Calculating the eigendecomposition of $A - B$, we find a negative eigenvalue -0.2 for the eigenvector $v=(\begin{smallmatrix} -\sqrt{2}/2 \\ \sqrt{2}/2 \end{smallmatrix})$. So, in the direction given by that eigenvector, $A$ has smaller variance than $B$, as the following calculation gives. If $X$ is a random variable with covar matrix $A$, and $Y$ has covar matrix $B$, then $$\DeclareMathOperator{\V}{\mathbb{V}} \V v^T X = v^T A v = 1.5 \\ \V v^T Y = v^T B v = 1.7 $$ so indeed, in that direction $A$ is smaller than $B$ despite that the overall impression is the opposite.