Vector-valued estimators, intuitively why $var(\widetilde{\beta})-var(\widehat{\beta})$ being p.s.d. means $\widehat{\beta}$ more efficient?

Question

For two scalar unbiased estimators $\widehat{\alpha}$ and $\widetilde{\alpha}$, we know that if one has smaller variance, then we say it is more efficient, which intuitively means that this estimator is more concentrated around true value (or has less dispersion). However, such an intuition seems to be lost in general for vector-valued estimators. For example, for vector-valued unbiased estimator $\widehat{\beta}$ and $\widetilde{\beta}$, we say that $\widehat{\beta}$ is more efficient than $\widetilde{\beta}$ if matrix $var(\widetilde{\beta})-var(\widehat{\beta})$ is positive semidefinite (p.s.d. in short), where $var(\widehat{\beta})$ and $var(\widetilde{\beta})$ denote variance covariance matrices.

I'm wondering how shall we intuitively interpret this positive semidefiniteness? Is there any intuitive connection between this positive semidefiniteness and $\widehat{\beta}$ being more concentrated just like for the scalar case? (Of course, if $var(\widehat{\beta})$ and $var(\widetilde{\beta})$ are both diagonal matrices, positive semidefiniteness means each element in $\widehat{\beta}$ has smaller variance. My question is about the more general case when they are not diagonal) Thanks!

Answer 1

Think about a vector $a$ specifying a linear combination of the $\hat\beta$s -- a direction in $\hat\beta$ space.

One natural way to extend the one-dimensional comparison based on variances is to say the linear combination $a\hat\beta$ has smaller variance than $a\tilde\beta$ for every such $a$. That means $$ a\mathrm{var}[\tilde\beta]a^T\geq a\mathrm{var}[\hat\beta]a^T$$ for any $a$, or $$a\left(\mathrm{var}[\tilde\beta]-\mathrm{var}[\hat\beta]\right)a^T\geq 0$$ which is just to say that the difference in the matrices is p.s.d.

Answer 2

This condition is a natural extension of the scalar variance inequality. The condition that $\mathbb{V}(\tilde{\boldsymbol{\alpha}})-\mathbb{V}(\hat{\boldsymbol{\alpha}})$ is positive definite is equivalent to saying that any linear combination of the elements of $\tilde{\boldsymbol{\alpha}}$ has higher variance than the same linear combination of the elements of $\hat{\boldsymbol{\alpha}}$. This latter framing of the condition is analogous to the scalar case that you already understand intuitively. To see this equivalence, note that if $\mathbf{A}$ is a positive definite matrix, then for any conformable vector $\mathbf{z}$ you have:

$$\mathbf{z}^\text{T} \mathbf{A} \mathbf{z} > 0.$$

Applying this to the positive definite matrix $\mathbb{V}(\tilde{\boldsymbol{\alpha}})-\mathbb{V}(\hat{\boldsymbol{\alpha}})$ you get:

$$\mathbb{V}(\mathbf{z} \cdot \tilde{\boldsymbol{\alpha}}) = \mathbb{V}(\mathbf{z}^\text{T} \tilde{\boldsymbol{\alpha}}) = \mathbf{z}^\text{T} \mathbb{V}(\tilde{\boldsymbol{\alpha}}) \mathbf{z} > \mathbf{z}^\text{T} \mathbb{V}(\hat{\boldsymbol{\alpha}}) \mathbf{z} = \mathbb{V}(\mathbf{z}^\text{T} \tilde{\boldsymbol{\alpha}}) = \mathbb{V}(\mathbf{z} \cdot \tilde{\boldsymbol{\alpha}}) .$$

This establishes that the variance of $\mathbf{z} \cdot \tilde{\boldsymbol{\alpha}}$ is higher than the variance of $\mathbf{z} \cdot \hat{\boldsymbol{\alpha}}$. In the scalar case there is only a single element, but in the vector case we are concerned with any linear combination of the elements of the vectors.