Can a Bayesian estimator perform better than an MVUE?

Question

According to wikipedia:

In statistics a minimum-variance unbiased estimator (MVUE) or uniformly minimum-variance unbiased estimator (UMVUE) is an unbiased estimator that has lower variance than any other unbiased estimator for all possible values of the parameter.

Can a biased Bayesian estimator perform better? If so, are there any examples?

I personally think it cannot since the least value of the (MSE = $\text{Bias}^2$ + Variance) is achieved by the MVUE, for a biased Bayesian estimator to perform better than this would mean that it has a strictly lower variance as compared to the MVUE. I'm unable to justify this part. Can a bayesian estimator have a lower variance than the MVUE?

Answer 1

First of all the equation is $MSE = Bias^2 + Variance$ and not Bias.

Now in MVUE estimators, the Bias is zero and the variance is equal to the CRLB (Cramer-Rao Lower Bound) calculated as follows:

CRLB($\hat \theta$) = $\dfrac{1}{-E[\dfrac{\partial^2 \ln P(X;\theta)}{\partial \theta^2}]}$

The MSE then is equal to the variance.

To define the Bayes estimators we need to define the cost function $C(\hat \theta,\theta)$ which is the cost of choosing $\hat \theta$ instead of $\theta$. Then the Bayes estimator is an estimator in which minimizes the expected cost function with respect to the posterior distribution. Mathematically it is defined as follows:

$E[C(\hat \theta,\theta)|x] = \int C(\hat \theta,\theta) P(\theta|x) d\theta$

Where $P(\theta|x)$ is the posterior distribution. Bayes estimator $\hat \theta$ minimizes the above expression.

Now if you let the cost to be the quadratic cost defined as follows:

$C(\hat \theta,\theta) = (\theta - \hat \theta)^2 $

Then the estimator becomes the MMSE (Minimum mean square estimator error) which minimizes the square of residuals or the MSE.

The MMSE estimators have the lowest possible MSE among all the estimators since they are designed to minimize the MSE and they are Bayesian estimators.

For more information and to better understand the difference between Bayesian (MMSE, LAE, and MAP) vs Frequentistis estimators (Maximum likelihood).please refer to this article

Essential Parameter Estimation Techniques in Machine Learning

Answer 2

The MVUE estimator (say $f$) achieves the lowest variance out of all estimators such that bias is zero. However, there is no reason why an estimator with non-zero bias (say $f'$) could not have a variance much lower than that of the MVUE, such that overall $\text{MSE}(f) > \text{MSE}(f')$ holds.

Consider for example the ridge regression estimator $f_\lambda$ with $\lambda \geq 0$ a regularisation parameter. Then $f_0$ is the least squares estimator which we know is a MVUE (Gauss-Markov theorem). However $$\inf_{\lambda} \text{MSE}(f_\lambda) \leq \text{MSE}(f_0),$$ by definition of an infimum, and frequently $f_\lambda$ can indeed achieve lower MSE for $\lambda > 0$.

Relating that to Bayesian estimators, ridge regression is equivalent to Bayesian linear regression with a Gaussian prior over the weights and a Gaussian likelihood.