12
$\begingroup$

Suppose I am estimating one of the parameter. Now if we plot the biased estimator of that and unbiased estimator of that can we say for sure that biased one has less variance than unbiased one always.

My thought: my analysis somehow tells me that it is true because if we check bias variance trade off then obviously if we increase the bias from 0 to something, that amount will be deduced from the variance part making the estimator peaky.

$\endgroup$
5
  • 12
    $\begingroup$ You say "the biased estimator of that" as if there is only one biased estimator of any parameter. This is not the case! Unless you restrict which estimators you are talking about, this is trivially false, and counterexamples are easily constructed. $\endgroup$
    – Glen_b
    Oct 1 '21 at 0:09
  • 7
    $\begingroup$ No. To illustrate @Glen_b's point, if you add 42 to the value obtained using the unbiased estimator, you would have a biased estimator with equal variance. If you then added some random noise to the estimate you would have a biased estimator with higher variance. I know this is a contrived thought experiment, but there is nothing in the theory of statistics that says estimators have to be sensible. $\endgroup$ Oct 1 '21 at 6:43
  • 3
    $\begingroup$ There is no the biased estimator and the unbiased estimator. There is a range of different estimators. A similar use of 'the' occurred in this question stats.stackexchange.com/questions/545471/… , which was about the unbiased estimator for a function of the parameter $\mu$ of a normal distributed population. However, when you speak about the minimum variance estimator, then there is indeed a unique solution (at least for the minimum variance unbiased estimator, I am not sure whether the biased case needs to be unique). $\endgroup$ Oct 1 '21 at 13:10
  • 1
    $\begingroup$ You likely are thinking about minimum-quadratic loss estimators, for which the question is useful (albeit simple to answer). $\endgroup$
    – whuber
    Oct 1 '21 at 18:56
  • $\begingroup$ While subtraction is one of the meanings of "deduced", "subtraction" is much confusing. $\endgroup$ Oct 2 '21 at 3:19
11
$\begingroup$

NO

Remember that just about anything can be an estimator, even silly estimators. Let’s consider two estimators for $k$ of $\chi^2_k$. Take an $iid$ sample $X_1,\dots,X_n$.

$$ \hat k_1=\bar X \\ \hat k_2=\sum_{i=1}^nX_i=n\bar X $$

$\hat k_1$ is unbiased, while $\hat k_2$ is biased. However, what are the variances?

$$ \mathbb{V}(\hat k_1)=2k/n\\ <\\ \mathbb{V}(\hat k_2)= 2nk \\ \forall n>1 $$

There’s your counterexample. $\square$

However, I see you making at least two mistakes in your setup.

  1. Multiple estimators can be unbiased. The sample mean, sample median, and first observation (NOT first order statistic) are unbiased estimators for the mean of a normal distribution, for example. (Indeed, the $j^{th}$ observation (NOT order statistic) drawn from a distribution is an unbiased estimator for the mean whenever the distribution has a mean.)

  2. The MSE does not have to be the same for biased and unbiased estimators. In fact, we tend to pick biased estimators over unbiased estimators because there is such a reduction in variance that the MSE decreases.

EDIT

An even easier example where we’re estimating $\mu$ of $N(\mu, \sigma^2)$:

$$ \hat\mu_1=\bar X\\ \hat\mu_2=\bar X+ 1 $$

Both have the same variance, yet only $\hat \mu_1$ is unbiased.

EDIT 2

If two estimators of a parameter $\theta$, one biased $(\hat\theta_1)$ by some amount $b$ and one unbiased $(\hat\theta_2)$, have the same $MSE$, then it must be that the biased estimator has lower variance. Let $MSE(\hat\theta_1) = MSE(\hat\theta_2) = M$.

$$ (\text{bias}(\hat\theta_1))^2 + \mathbb{Var}(\hat\theta_1) = M = (\text{bias}(\hat\theta_2))^2 + \mathbb{Var}(\hat\theta_2) $$$$ b^2 + \mathbb{Var}(\hat\theta_1) = 0 + \mathbb{Var}(\hat\theta_2) $$$$ \mathbb{Var}(\hat\theta_1) = \mathbb{Var}(\hat\theta_2) - b^2$$$$ \implies\mathbb{Var}(\hat\theta_1) > \mathbb{Var}(\hat\theta_2) $$

(I confess that I don't know what happens if we work over complex numbers (so that $b^2<0$ is possible), though I would imagine that the decompision of $MSE$ in that case is something like $MSE(\hat\theta) = (\text{bias}(\hat\theta)) \overline{(\text{bias}(\hat\theta))} + \mathbb{Var}(\hat\theta)$.)

$\endgroup$
9
  • $\begingroup$ The second point is completely new to me. $\endgroup$
    – user27286
    Oct 1 '21 at 5:18
  • $\begingroup$ Then it sounds like a good next question to post! $\endgroup$
    – Dave
    Oct 1 '21 at 11:49
  • $\begingroup$ if you know that two estimators have the same MSE and one is biased, then it does follow that the biased one has lower variance, because MSE = squared bias + variance. Is that the source of the confusion? $\endgroup$
    – JDL
    Oct 1 '21 at 14:02
  • $\begingroup$ @JDL Yes yes, it started from there but then somehow stupidly I imagined all estimators will have the same MSE and then concluded that. It's not the case. So you can draw two estimators in a graph and then whenever you see a peaky one, doesn't mean that it will be the biased one, this is a wrong thing to think unless of course, you know that both of the estimators have same MSE. $\endgroup$
    – user27286
    Oct 1 '21 at 15:27
  • $\begingroup$ @user27286 You have accepted my answer, so I am inclined not to modify it. Would you like me to add some comments about what happens if we know that two estimators have the same MSE? $\endgroup$
    – Dave
    Oct 1 '21 at 16:08
9
$\begingroup$

There are many, many, many different possible estimators in estimation problems. In general there are multiple unbiased estimators and multiple biased estimators, and their variances need to be considered on a case-by-case basis. A biased estimator may have a lower, or higher, or the same variance as an unbiased estimator. Now, if we consider estimators on the basis of their mean-squared-error (MSE), then we will generally consider the lower bias and higher variance as a trade-off and we will ignore any estimators that have both higher bias and higher variance than another estimator. In this context we are left with estimators where the unbiased ones have higher variance and the biased ones have lower variance. Nevertheless, this is a contextual result that comes from setting other estimators out of consideration.


As a counter-example to show that it is possible to have an unbiased estimator with higher variance than a biased estimator, consider a set of data values $X_1,...,X_n \sim \text{N}(\mu, 1)$ and consider the estimators:

$$\hat{\mu}_1 \equiv X_1 + b \quad \quad \quad \quad \quad \hat{\mu}_2 \equiv \bar{X}_n,$$

where $b \neq 0$. It is simple to establish that $\hat{\mu}_1$ is biased and $\hat{\mu}_2$ is unbiased, and their respective variances are:

$$\mathbb{V}(\hat{\mu}_1) = 1 \quad \quad \quad \quad \quad \mathbb{V}(\hat{\mu}_2) = \frac{1}{n}.$$

If $n>1$ (i.e., if you have more than one data point) then the first estimator has a higher variance. Since this is the biased estimator, it is both biased and has higher variance than the unbiased estimator.

$\endgroup$
3
  • $\begingroup$ I think we have a typo in the line ` multiple unbiased estimators and multiple unbiased estimators,`. But well I think one key take way that I think I can take from your answer is the part that "if two estimators have same MSE then the biased one will have lower variance than the unbiased one". Correct me if I am wrong. $\endgroup$
    – user27286
    Oct 1 '21 at 8:56
  • $\begingroup$ I might change $\epsilon$ to something like $c$, since $\epsilon$ in statistics so often represents a random variable. $\endgroup$
    – Dave
    Oct 1 '21 at 11:46
  • 1
    $\begingroup$ @Dave: I changed it to $b$ for bias. $\endgroup$
    – Ben
    Oct 1 '21 at 21:53
4
$\begingroup$

I don't think there's an answer as to which will have more variance. Take the case of the sample variance. The unbiased estimate

is $\sum_i (x_{i} - \bar{x})^2/(n-1)$. It has variance

$\left(\frac{1}{n-1}\right)^2$ Var$\left(\sum_i (x_{i} - \bar{x})\right)^2$

The biased estimate has variance:

$\left(\frac{1}{n}\right)^2$ Var$\left(\sum_i (x_{i} - \bar{x})\right)^2$

So, in this case, the biased estimate will have less variance. Each case probably depends on the estimate and a generalization as to which has or more less variance is probably not possible.

$\endgroup$
3
  • $\begingroup$ What you call "the" biased estimator is presumably $\sum_i (x_{i} - \bar{x})^2/n$. Meanwhile $\sum_i (x_{i} - \bar{x})^2/(n+1)$ is also biased but has a lower variance and expected mean-square error than either of those, and $\sum_i (x_{i} - \bar{x})^2/(n+2)$ has an even lower variance but higher mean-square error, while $\sum_i (x_{i} - \bar{x})^2/(n-2)$ is also biased but has a higher variance and expected mean-square error $\endgroup$
    – Henry
    Oct 1 '21 at 17:53
  • $\begingroup$ @Henry: I agree with everything you say and it's consistent with my last statement that the variance will generally depend on the estimate one uses so there's no general result that says that one estimate ( unbiased or biased ) will have less ( or more ) variance than another. It just depends on the particular comparison one is making. $\endgroup$
    – mlofton
    Oct 1 '21 at 20:40
  • $\begingroup$ @Henry: Note that I just used $n$ and $n-1$ because those are the commonly used denominators in the sample variance expression. $\endgroup$
    – mlofton
    Oct 1 '21 at 20:42
2
$\begingroup$

Yes

Provided, that the bias is decreasing the variance of the error.

In that case, since the bias reduces the variance of the error, which can be decomposed into contributions from bias and variance, the variance must be decreasing.

$$\text{var}(error) = bias(estimator)^2 + \text{var}(estimator)$$

So if you reduce $\text{var}(error)$ while increasing $bias(estimator)^2$ then necessarily $\text{var}(estimator)$ must decrease.

This means that the only sensible types of bias (sensible meaning that it reduces the error) are the ones that reduce the variance of the estimator. As explained below this is not true for every type of biased estimator.


No

However, if one uses some silly estimator with enormous variance and that has also bias, then sure a biased estimator can have more variance.

Example, related question

This is related to the question

Why exactly $E[(\hat{\theta}_n - E[\hat{\theta}_n])^2]$ and $E[\hat{\theta}_n - \theta]$ cannot be decreased simultaneously?

Seen here: Bias / variance tradeoff math

In an answer to that question we see the following graph for a shrinking estimator.

We can estimate the mean of some population $\mu$ by the sample mean $\bar{x}$ and then multiply it with some scaling factor $c\bar{x}$.

In the graph on the right you see what happens with variance and bias when the shrinking (or inflating) parameter is changed. The unbiased estimator is in the middle ($c=1$). You can add bias by multiplying with a factor below one ($c<1$to the left of the graph) or with a factor above one ($c>1$to the right of the graph).

You can see that multiplying with a factor above one is not decreasing the variance of the estimator (obviously since the variance of the estimator scales with $c^2$).

But, this is also not a type of bias that decreases the variance of the error, and it is not a type of bias that is typically considered in a bias-variance trade-off.

overfitting and underfitting in shrinking of sample mean

$\endgroup$
5
  • 2
    $\begingroup$ That “yes” has a gigantic caveat, more or less, “Yes, it happens when it happens,” but as multiple answers and comments show, it doesn’t have to happen. $\endgroup$
    – Dave
    Oct 1 '21 at 11:44
  • $\begingroup$ @Dave I am aware of the caveat and it is discussed with a lot of detail in the rest of the post. Especially for you and other people that don't read further and try to understand the issue (and haphazardly downvote questions while doing so) I have stressed the very important sentence that follows the yes and specifies the conditions when it is a yes. $\endgroup$ Oct 1 '21 at 12:36
  • $\begingroup$ @Dave you can be nitpicking about the fact that biased estimators are not necessarily decreasing the variance of the estimator. But in that case you are missing the point or line of thought of the OP who is considering the case of bias-variance trade-off. There is no good use in telling the OP that the answer is no and that he is simply wrong. It is better to tell the OP in what special case the answer is indeed yes since that is what the OP was intuitively thinking about. $\endgroup$ Oct 1 '21 at 12:49
  • $\begingroup$ Assuming bias and errors are uncorrelated? $\endgroup$
    – Aksakal
    Oct 1 '21 at 12:49
  • $\begingroup$ @Aksakal I see the bias as the expectation of the error. Can this be correlated with the errors? $\endgroup$ Oct 1 '21 at 12:52
0
$\begingroup$

It is certainly not the case that all biased estimators have less variance than unbiased ones. That is quite trivial to see: pick any unbiased estimator, then create a new estimator that consists of that estimator plus some random non-zero mean random variable. It is also not the case the lowest variance biased estimator always has lower variance than the lowest variance unbiased estimator. For instance, given standard linear regression assumptions, OLS gives a zero bias estimator that has the lowest overall variance.

However, if the lowest variance unbiased estimator has a lower variance than the lowest variance biased estimator, then there probably isn't any reason to use the biased estimator, so you aren't going to hear about the biased estimator. That is, if estimator with the lowest variance overall is unbiased, then people are going to just use that one, and there's no reason to examine which biased estimators have the lowest variance. So ... I guess you can say that you have a biased view of biased estimators, because you only hear about the ones that have lower variance than the unbiased estimator.

$\endgroup$
2
  • $\begingroup$ "OLS gives a zero bias estimator that has the lowest overall variance." It is unclear what you mean by 'lowest overall variance'. Do you mean 'lowest mean squared error'? And does OLS really give the lowest error? Are you referring to the specific case of unbiased linear estimators, ie the Gauss Markov theorem? In that case it is not a counterexample because the OLS estimator giving the lowest variance (but only compared to other unbiased estimators) does not exclude biased estimators having lower mean squared error. $\endgroup$ Oct 2 '21 at 12:46
  • $\begingroup$ The Gauss-Markov theorem also makes the minimum-variance claim for linear estimators. If we allow nonlinear estimators, we can have unbiased estimators with smaller variance. (I was wrong in my question when I assumed that the quantile/median regression estimator is linear. Hardy explains in the answer and comments.) $\endgroup$
    – Dave
    Oct 2 '21 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.