0
$\begingroup$

Suppose that $\mu$ is an unknown $k$-vector, which one seeks to guess. You observe $n$ i.i.d. normal variates $x_i \sim \mathcal{N}\left(\mu,\Sigma\right)$, say stacked in the matrix $X$, then produce estimate $\hat{\mu}\left(X\right)$. Cramer-Rao theorem gives a lower bound on $Var\left(\hat{\mu}\left(X\right)\right)$ when $\hat{\mu}$ is unbiased.

However, the variance (and MSE) for a biased $\hat{\mu}$ might be lower than an unbiased one. You might be tempted to choose a biased estimator. My question is whether there is a "free lunch" type argument against a biased estimator. Suppose that one will draw $\mu$ uniformly from all $k$-vectors with unit norm, then generate the $X$, and compute the estimate. Is there a lower bound for $$ E_{\mu}\left[E_{X} \left[\left(\hat{\mu}\left(X\right) - \mu\right)^2\right] \right] $$ that applies to all estimators?

$\endgroup$
1
  • 1
    $\begingroup$ I'm not sure to get the question right, but the Cramer-Rao bound is still valide for a biased estimator. For a one dimensionnal estimator, you have $E((\hat\mu(X) - \mu)^2) \geq (1 + bias^2) / J$ where J is the Fisher information such that $E_\mu[E((\hat\mu(X) - \mu)^2)] \geq E_\mu[(1 + bias^2) / J]$. $\endgroup$
    – Pohoua
    Jan 21, 2021 at 8:38

1 Answer 1

2
$\begingroup$

If one picks a probability distribution $\pi$ on $\mu$ and minimises the integrated risk, this is equivalent to running a Bayesian decision analysis with this prior distribution. In this case the optimal solution is the posterior mean $$\hat\mu(x)=\mathbb E^\pi[\mu|x]$$ and the estimator is always biased. There is thus gains to be found by reaching outside unbiasedness.

$\endgroup$
3
  • $\begingroup$ I'm ashamed to say I am not familiar with Bayesian decision analysis, but this looks to be just what I want. Am reading some references now. Thanks, $\endgroup$
    – shabbychef
    Jan 22, 2021 at 6:38
  • $\begingroup$ Is Berger's Statistical Decision Theory still the right text for an intro to the topic? $\endgroup$
    – shabbychef
    Jan 27, 2021 at 6:58
  • $\begingroup$ Yes, Jim's book is completely appropriate for such an introduction. $\endgroup$
    – Xi'an
    Jan 27, 2021 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.