# Expected variance of biased estimators

Suppose that $$\mu$$ is an unknown $$k$$-vector, which one seeks to guess. You observe $$n$$ i.i.d. normal variates $$x_i \sim \mathcal{N}\left(\mu,\Sigma\right)$$, say stacked in the matrix $$X$$, then produce estimate $$\hat{\mu}\left(X\right)$$. Cramer-Rao theorem gives a lower bound on $$Var\left(\hat{\mu}\left(X\right)\right)$$ when $$\hat{\mu}$$ is unbiased.

However, the variance (and MSE) for a biased $$\hat{\mu}$$ might be lower than an unbiased one. You might be tempted to choose a biased estimator. My question is whether there is a "free lunch" type argument against a biased estimator. Suppose that one will draw $$\mu$$ uniformly from all $$k$$-vectors with unit norm, then generate the $$X$$, and compute the estimate. Is there a lower bound for $$E_{\mu}\left[E_{X} \left[\left(\hat{\mu}\left(X\right) - \mu\right)^2\right] \right]$$ that applies to all estimators?

• I'm not sure to get the question right, but the Cramer-Rao bound is still valide for a biased estimator. For a one dimensionnal estimator, you have $E((\hat\mu(X) - \mu)^2) \geq (1 + bias^2) / J$ where J is the Fisher information such that $E_\mu[E((\hat\mu(X) - \mu)^2)] \geq E_\mu[(1 + bias^2) / J]$. Jan 21, 2021 at 8:38

If one picks a probability distribution $$\pi$$ on $$\mu$$ and minimises the integrated risk, this is equivalent to running a Bayesian decision analysis with this prior distribution. In this case the optimal solution is the posterior mean $$\hat\mu(x)=\mathbb E^\pi[\mu|x]$$ and the estimator is always biased. There is thus gains to be found by reaching outside unbiasedness.