5
$\begingroup$

In my Bayesian statistics class, my professor makes the remark that we should not be obsessed with unbiased estimator.

First: I understand this statement in the sense of trading biasedness for smaller variance, i.e. "good" estimator = estimator with smallest Mean Squared Error (MSE). This idea is seen among Frequentists also.

But second, and the confusing part: My professor then said something about an unbiased estimator is picking up noise and not signal. Since most of the observed data are just noise, we want to shrink our estimate to 0.

I do not quite understand what is noise and signal: 1) Don't all data points come from the same distribution, with the difference due only to sampling variation? 2) Why can't unbiased estimator (say, sample mean of a normal model) differentiate noise and signal? If it can't, what sort of estimator can?

An example would be great, and a social science example would be best!

$\endgroup$
4
$\begingroup$

Some people are annoyingly obsessed with unbiasedness. Bias and dispersion are measures of uncertainty which roughly correspond to accuracy and precision. You usually have a trade-off between accuracy and precision, some estimators may be more precise but less accurate and vice versa.

MSE is a sum of bias and the variance: $MSE=E[\beta-\hat{\beta}]^2=E[\beta-E[\hat{\beta}]+E[\hat{\beta}]-\hat{\beta} ]^2=bias^2+Var[\hat{\beta}]$

Example. Let's say your parameter is from Breit Wiegner distribution. enter image description here

There is no unbiased estimator. The bias is defined as $bias=E[\hat{\beta}]-\beta$, where $\beta$ - true value, and $\hat{\beta}$ is its estimator. In this case $E[\hat{\beta}]$ is not defined mathematically, so you can't compute the bias.

This is an extreme example where unbiased estimator does not exist at all.

$\endgroup$
  • 1
    $\begingroup$ I believe there are many unbiased estimators of the center of this distribution: the median is one. You might be thinking of unbiased linear estimators. $\endgroup$ – whuber Apr 18 '14 at 22:14
  • $\begingroup$ The median work fine, but you can't compute its bias in the usual definition because it requires knowing $E(\hat{\beta})$ which isn't mathematically defined. Median is an optimal estimator in some sense but not in MSE minimization regard. $\endgroup$ – Aksakal Apr 18 '14 at 22:17
  • 1
    $\begingroup$ For sample sizes of $3$ or greater, the bias of the median does have an expectation and it is zero. The "usual definition" I am using is the difference between the expectation of an estimator and the value of its estimand. (For sample sizes of $5$ or more, it even has a variance. E.g., the variance of the median when $n=5$ is $90\zeta(3)/\pi^4-1/2\approx 0.6106$.) $\endgroup$ – whuber Apr 18 '14 at 22:24
  • $\begingroup$ Ok, you are talking about the center of the distribution, then you right. I meant the expectation. $\endgroup$ – Aksakal Apr 18 '14 at 22:27
  • 1
    $\begingroup$ I thought you might have that in mind--but then you're in even worse trouble, because you are trying to talk about estimators of something that doesn't even exist! That's not really relevant to the existence (or lack thereof) of biased estimators of actual parameters. $\endgroup$ – whuber Apr 18 '14 at 22:30
3
$\begingroup$

The motivation of the statement is a rejection of some very important work from Rao-Blackwell about the provision of so called UMVUEs (Uniform minimum variance unbiased estimators). This showed that there is a lower bound on variance of unbiased estimators, and if you achieve that, you usually get a very good estimator. Many of our most popular statistics are special cases of that, such as the z-test for normal data, or the least squares regression models (among linear estimators, which makes LS BLUE (best linear unbiased estimator)).

The big question is what happens when we stop caring about bias? Do we get better estimators? The answer is YES. For inference, plenty of biased estimators are extremely efficient at detecting associations in data. A great way of evaluating the quality of an estimator is with its MSE (mean squarred error), or:

$\mbox{MSE}(\hat{\theta}) = \mathcal{E}\left( (\hat{\theta} - \theta)^2\right)$

This can be rewritten as:

$\mbox{MSE}(\hat{\theta}) = \mathcal{E}\left( (\mathcal{E}(\hat{\theta}) - \theta)^2\right) + \mathcal{E} \left( (\mathcal{E}(\hat{\theta}) - \hat{\theta} )^2 \right)$

which is the sum of the squared bias and the variance of the estimator.

An example of an estimator that has better MSE than the UMVUE for a multivariate normal model is ridge regression. What your professor may have referred to regarding "shrink to zero" is shrinkate: the tendency for high dimensional statistics to have poor out-of-sample validity. The "squared error loss" (which is variance plus squared-bias) can be improved from using multivariate normal MLE. The ridge estimator uses an L2 penalty to penalize highly variable estimates. The LASSO uses an L1 penalty and also has shrinkage properties. The L1 penalty forces relatively small estimates to be exactly zero, but it is an abuse of terminology to claim they "shrink to zero" when shrinkage refers to optimizing MSE. Rather, we constrain.

http://www.few.vu.nl/~wvanwie/Courses/HighdimensionalDataAnalysis/WNvanWieringen_HDDA_Lecture234_RidgeRegression_20182019.pdf

Bayes estimators which are posterior means using a conjugate prior minimize squared error loss.

Bearing that in mind, if MSE was all we called about controlling for, you can come up with a plethora of estimators which are a little biased but with much smaller variance than the UMVUE. And given that UMVUE estimators can have complicated asymptotic distributions, or are sometimes difficult to find, or sometimes don't exist, we often seek out biased estimators that lack all those problems.

$\endgroup$
  • $\begingroup$ Thanks for your answer, but this is what I already understood (as indicated in the first part of my question). $\endgroup$ – Heisenberg Apr 18 '14 at 20:44
  • 1
    $\begingroup$ Good point. I would ask for clarification on that point. If he's talking about the so-called class of "robust" and/or "trimmed" estimators, then these are estimators which discard, say, 5% of positive/negative tails. So, in the case of model misspecification (e.g. non-central Cauchy location estimation), they make much better estimates of location than sample averages. Otherwise, I don't think there's much credibility to that statement. $\endgroup$ – AdamO Apr 18 '14 at 21:23
  • $\begingroup$ I guess another way to phrase what he said is: Bayesian estimation uses shrinkage, and shrinkage is a good thing. Why is it? $\endgroup$ – Heisenberg Apr 18 '14 at 21:28
  • $\begingroup$ Thanks again for the clarification. My understanding is that if you have a non-informative proper prior, the Bayes estimate minimizes the MSE. For instance, in the estimation of a population proportion, a beta 1 1 prior (flat) will give you a beta posterior estimate. The Bayes' estimate (posterior mean) is a minimal MSE estimator. My understanding is that "shrinking" occurs even when the prior is non-informative simply because you have readily assumed a probability model for the parameter (rather than treating it as fixed). $\endgroup$ – AdamO Apr 18 '14 at 21:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.