# Multivariate Cramér-Rao inequality: intuition for positive semidefiniteness

Here's what Wikipedia says about the Multivariate Cramér-Rao inequality:

If $\boldsymbol{T}(X)$ is an unbiased estimator of $\boldsymbol{\theta}$, then the Cramér–Rao bound reduces to $\mathrm{cov}_{\boldsymbol{\theta}}\left(\boldsymbol{T}(X)\right) \geq I\left(\boldsymbol{\theta}\right)^{-1}$.

The matrix inequality $A \ge B$ is understood to mean that the matrix $A-B$ is positive semidefinite.

I understand everything above. But I started playing around with examples and came up with something that doesn't make sense to me. Suppose we have two unbiased estimators of some two-dimensional $\boldsymbol{\theta}$, one with covariance matrix

$A=\left[\begin{array}{cc} 3 & 1.5\\ 1.5 & 3 \end{array}\right]$

and the other with covariance matrix

$B=\left[\begin{array}{cc} 2 & 0.3\\ 0.3 & 2 \end{array}\right]$.

Now $A-B$ is not positive semidefinite. So even though each parameter estimate has a lower variance in $B$, and there's less covariance between the two of them, $B$ doesn't 'count' as having lower variance than A in the psd sense.

Can someone give an intuitive explanation why? (I guess maybe I'm really asking why is the generalized inequality with respect to the psd cone the 'right' comparison here? Or is it just the only one for which we can prove this result?)

• When the difference of two covariance matrices is not definite, it means that some linear combinations will have larger variance with one of them, other linear combinations will have larger variance with the other one. So there is no domination. Feb 9, 2015 at 8:34
• Thanks. But: why does it matter about linear combinations? And can you give an example of such a linear combination in this case? (I'm having trouble seeing one.)
– mww
Feb 10, 2015 at 1:07
• For an example, take the diagonal case: $A=\begin{smallmatrix} 2&0\\0&3\end{smallmatrix}, B=\begin{smallmatrix}3&0\\0&2\end{smallmatrix}$. Then the linear comb of variables $(X,Y)$ taken with that covariance matrix, with coefficients $(1,0)$ has bigest variance for $B$, while lincomb with coef $(0,1)$ has bigest varaince for $A$. Feb 11, 2015 at 18:15
• But surely my examples of A and B don't present that problem?
– mww
Feb 11, 2015 at 19:47
• $A$ has an eigenvalue, $3/2$, which is smaller than the smallest eigenvalue of $B$ (which is $17/10$). Thus ellipses defined by $x^\prime A x=c$ are actually a little narrower than ellipses $x^\prime B x = c$. Equivalently, there exist linear combinations of the (implied) underlying random variables which have a smaller variance for $A$ than for $B$.
– whuber
Jun 5, 2017 at 19:17

## 2 Answers

I find it useful to think of the Convolution Theorem: a large-sample independent-observation version.

It says that (asymptotically), for any regular estimator $$T$$ we have $$T(X)-\theta= Z+ \Delta$$ where $$Z$$ has covariance matrix $$I^{-1}(\theta)$$ and $$\Delta$$ is independent of $$Z$$. That is, any estimator is the efficient estimator plus pure noise. Which is a nice property for an efficiency bound to have. [Another version says this holds for any estimator $$T(X)$$ at almost all $$\theta$$]

If you look at what this says about $$\mathrm{cov}[T(X)]$$, it is exactly that $$\mathrm{cov}[T(X)]\geq I^{-1}(\theta)$$

Things are more complicated in small samples-- for example, there need not be a fully efficient estimator -- but the large-sample motivation is still relevant.

So the covariance matrices are $$A=(\begin{smallmatrix} 3 & 1.5 \\ 1.5 & 3\end{smallmatrix})$$ and $$B=(\begin{smallmatrix} 2 & 0.3 \\ 0.3 2\end{smallmatrix})$$. Yes, visually it seems like $$A$$ is "bigger than $$B$$, and really, that holds for the determinants "generalized variances" which are 6.75 and 3.91. But that impression is deceiving. Calculating the eigendecomposition of $$A - B$$, we find a negative eigenvalue -0.2 for the eigenvector $$v=(\begin{smallmatrix} -\sqrt{2}/2 \\ \sqrt{2}/2 \end{smallmatrix})$$. So, in the direction given by that eigenvector, $$A$$ has smaller variance than $$B$$, as the following calculation gives. If $$X$$ is a random variable with covar matrix $$A$$, and $$Y$$ has covar matrix $$B$$, then $$\DeclareMathOperator{\V}{\mathbb{V}} \V v^T X = v^T A v = 1.5 \\ \V v^T Y = v^T B v = 1.7$$ so indeed, in that direction $$A$$ is smaller than $$B$$ despite that the overall impression is the opposite.