Here's what Wikipedia says about the Multivariate Cramér-Rao inequality:

If $\boldsymbol{T}(X)$ is an unbiased estimator of $\boldsymbol{\theta}$, then the Cramér–Rao bound reduces to $\mathrm{cov}_{\boldsymbol{\theta}}\left(\boldsymbol{T}(X)\right) \geq I\left(\boldsymbol{\theta}\right)^{-1}$.

The matrix inequality $A \ge B$ is understood to mean that the matrix $A-B$ is positive semidefinite.

I understand everything above. But I started playing around with examples and came up with something that doesn't make sense to me. Suppose we have two unbiased estimators of some two-dimensional $\boldsymbol{\theta}$, one with covariance matrix

$A=\left[\begin{array}{cc} 3 & 1.5\\ 1.5 & 3 \end{array}\right]$

and the other with covariance matrix

$B=\left[\begin{array}{cc} 2 & 0.3\\ 0.3 & 2 \end{array}\right]$.

Now $A-B$ is not positive semidefinite. So even though each parameter estimate has a lower variance in $B$, and there's less covariance between the two of them, $B$ doesn't 'count' as having lower variance than A in the psd sense.

Can someone give an intuitive explanation why? (I guess maybe I'm really asking why is the generalized inequality with respect to the psd cone the 'right' comparison here? Or is it just the only one for which we can prove this result?)

  • $\begingroup$ When the difference of two covariance matrices is not definite, it means that some linear combinations will have larger variance with one of them, other linear combinations will have larger variance with the other one. So there is no domination. $\endgroup$ – kjetil b halvorsen Feb 9 '15 at 8:34
  • $\begingroup$ Thanks. But: why does it matter about linear combinations? And can you give an example of such a linear combination in this case? (I'm having trouble seeing one.) $\endgroup$ – mww Feb 10 '15 at 1:07
  • 1
    $\begingroup$ For an example, take the diagonal case: $A=\begin{smallmatrix} 2&0\\0&3\end{smallmatrix}, B=\begin{smallmatrix}3&0\\0&2\end{smallmatrix}$. Then the linear comb of variables $(X,Y)$ taken with that covariance matrix, with coefficients $(1,0)$ has bigest variance for $B$, while lincomb with coef $(0,1)$ has bigest varaince for $A$. $\endgroup$ – kjetil b halvorsen Feb 11 '15 at 18:15
  • $\begingroup$ But surely my examples of A and B don't present that problem? $\endgroup$ – mww Feb 11 '15 at 19:47
  • $\begingroup$ $A$ has an eigenvalue, $3/2$, which is smaller than the smallest eigenvalue of $B$ (which is $17/10$). Thus ellipses defined by $x^\prime A x=c$ are actually a little narrower than ellipses $x^\prime B x = c$. Equivalently, there exist linear combinations of the (implied) underlying random variables which have a smaller variance for $A$ than for $B$. $\endgroup$ – whuber Jun 5 '17 at 19:17

So the covariance matrices are $A=(\begin{smallmatrix} 3 & 1.5 \\ 1.5 & 3\end{smallmatrix})$ and $B=(\begin{smallmatrix} 2 & 0.3 \\ 0.3 2\end{smallmatrix})$. Yes, visually it seems like $A$ is "bigger than $B$, and really, that holds for the determinants "generalized variances" which are 6.75 and 3.91. But that impression is deceiving. Calculating the eigendecomposition of $A - B$, we find a negative eigenvalue -0.2 for the eigenvector $v=(\begin{smallmatrix} -\sqrt{2}/2 \\ \sqrt{2}/2 \end{smallmatrix})$. So, in the direction given by that eigenvector, $A$ has smaller variance than $B$, as the following calculation gives. If $X$ is a random variable with covar matrix $A$, and $Y$ has covar matrix $B$, then $$\DeclareMathOperator{\V}{\mathbb{V}} \V v^T X = v^T A v = 1.5 \\ \V v^T Y = v^T B v = 1.7 $$ so indeed, in that direction $A$ is smaller than $B$ dispite that the overall impression is the opposite.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.