1
$\begingroup$

Suppose that $\mu$ is an unknown $k$-vector, which one seeks to guess. You observe $n$ i.i.d. normal variates $x_i \sim \mathcal{N}\left(\mu,\Sigma\right)$, say stacked in the matrix $X$, then produce estimate $\hat{\mu}\left(X\right)$. Cramer-Rao theorem gives a lower bound on $Var\left(\hat{\mu}\left(X\right)\right)$ when $\hat{\mu}$ is unbiased.

However, the variance (and MSE) for a biased $\hat{\mu}$ might be lower than an unbiased one. You might be tempted to choose a biased estimator. My question is whether there is a "free lunch" type argument against a biased estimator. Suppose that one will draw $\mu$ uniformly from all $k$-vectors with unit norm, then generate the $X$, and compute the estimate. Is there a lower bound for $$ E_{\mu}\left[E_{X} \left[\left(\hat{\mu}\left(X\right) - \mu\right)^2\right] \right] $$ that applies to all estimators?

$\endgroup$
1
  • 1
    $\begingroup$ I'm not sure to get the question right, but the Cramer-Rao bound is still valide for a biased estimator. For a one dimensionnal estimator, you have $E((\hat\mu(X) - \mu)^2) \geq (1 + bias^2) / J$ where J is the Fisher information such that $E_\mu[E((\hat\mu(X) - \mu)^2)] \geq E_\mu[(1 + bias^2) / J]$. $\endgroup$
    – Pohoua
    Commented Jan 21, 2021 at 8:38

1 Answer 1

2
$\begingroup$

If one picks a probability distribution $\pi$ on $\mu$ and minimises the integrated risk, this is equivalent to running a Bayesian decision analysis with this prior distribution. In this case the optimal solution is the posterior mean $$\hat\mu(x)=\mathbb E^\pi[\mu|x]$$ and the estimator is always biased. There is thus gains to be found by reaching outside unbiasedness.

$\endgroup$
3
  • $\begingroup$ I'm ashamed to say I am not familiar with Bayesian decision analysis, but this looks to be just what I want. Am reading some references now. Thanks, $\endgroup$
    – shabbychef
    Commented Jan 22, 2021 at 6:38
  • $\begingroup$ Is Berger's Statistical Decision Theory still the right text for an intro to the topic? $\endgroup$
    – shabbychef
    Commented Jan 27, 2021 at 6:58
  • $\begingroup$ Yes, Jim's book is completely appropriate for such an introduction. $\endgroup$
    – Xi'an
    Commented Jan 27, 2021 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.