1
$\begingroup$

I'm quite confused about the definition of the bias of an estimator.

Suppose we have unknown distribution $P(x, \theta)$, and construct the estimator $\hat{\theta}$ that maps the observed data sample to values that we hope are close to $\theta$. Then the bias of an estimator is defined by:

$$Bias(\hat{\theta}) = \mathbb{E}(\hat{\theta}) - \theta ,$$

How can we calculate the $Bias(\hat{\theta})$ if we know neither the parent distribution $P(x, \theta)$ nor the true value of parameter $\theta$?

Is there a need for any assumptions?


upd: suppose I've proved, that estimator of the mean ($\hat{\mu}=\frac{1}{m}\sum_{i=1}^mx_i$) is unbiased for the Gaussian distribution, but is there any distribution for which the same estimator is biased? Or is it possible to prove such an estimator is unbiased for any distribution?

I guess such a question is caused by the lack of intuition about estimator and its bias. May be there is some convenient way to think about it?

$\endgroup$
2
  • $\begingroup$ Example: If $X_i \stackrel{iid}{\sim}\mathsf{Norm}(\mu,\sigma),$ then $E[\frac 1n\sum_{i=1}^n (X_i - \bar X)^2] = \frac{n-1}{n}\sigma^2< \sigma^2,$ even if $\sigma^2>0$ unknown. $\endgroup$
    – BruceET
    Aug 8 '21 at 16:32
  • $\begingroup$ Of course, you need to make assumptions $\endgroup$
    – Aksakal
    Aug 8 '21 at 17:40
1
$\begingroup$

When we ask if an estimator is unbiased, it is important to add that it is unbiased for a given quantity $\theta$, so I would add that to the definition. On the calculation, you do the computations under the assumptions for the distribution you are working. It may have a parametric form or not.

To exemplify, let $F$ be a c.d.f. (playing the role of your $P(x, \theta)$), and assume we have an i.i.d. sample $(X_i)_{i=1}^n$ such that $X_i \sim F$. The only hypothesis we will assume is that $\theta = E[X_1] = \int xdF$ exists. Notice something very important: the distribution $F$ is not parametrized by $\theta$. Indeed, our estimators are not necessarily for "parameters", but rather for functions of your distribution (usually functionals). For example, there are problems in which you are interested in estimating the density of $F$ (the p.d.f.), and you do not think of it as a "parameter" of $F$.

Now, lets show that $\hat{\theta} = \frac{1}{n} \sum_{i=1}^n X_i$ is unbiased for $\theta$.

$$ E\left[\widehat{\theta}\right] = \frac{1}{n} E\left[\sum_{i=1}^n X_i\right] = \frac{1}{n} \sum_{i=1}^n E\left[X_i\right] = \frac{1}{n} \sum_{i=1}^n\int xdF = \theta \quad.$$

We only used properties of the expectation, which hold for any $F$, even if we do not know its "exact form".

Have we proved then that the sample mean is unbiased for any statistical model? Not at all. For instance, $\theta$ might not be well defined! If $F$ ranges over a Cauchy family, then it does not make sense to estimate $\theta$.

A similar problem might arise if the sample is not i.i.d.: $\theta$ may not be defining exactly what you want, and even if it is, the estimator might not be unbiased. For example, suppose that $(Y_i)_{i=1}^n$ are i.i.d. with $Y_i \sim \mathcal{N}(\theta, 1)$, but your sample is $\mathbf{X} = (X_i)_{i=1}^n$ with $X_i = (-1)^{n+1} Y_i$. If you want to estimate $\theta$, you actually have $\theta = E[X_1]$, so it is defined. However, the sample mean is not unbiased, as you can easily verify.

$\endgroup$
0
$\begingroup$

Think of the bias as "if I repeat my experiment many times and apply my estimator each time, how much does the average estimation result differ from the true result?"

For example, the sample variance $$ \hat{\sigma}_Y^2 = \frac{1}{N}\sum_{i=1}^N (Y_i-\bar{Y})^2 = \frac{N-1}{N} \sigma_Y^2 $$ is a biased estimator of the variance of a distribution, which means that on average over many repeated experiments it will under-estimate the true variance $\sigma_Y^2$. This does not mean that it will under-estimate it every single time.

To assess the biasedness or unbiasedness of the estimator, one will usually make some assumptions about $P(x, \theta)$. E.g., the derivation of the biasedness of the sample variance assumes independent and identically distributed (i.i.d.) measurements. The magnitude of the bias can in general depend on many factors such as the sample size $N$ and the true parameter vector $\theta$.

To put it another way, the bias of an estimator is something that is usually analyzed theoretically under specific assumptions. Therefore, one can assume full knowledge of the true parameters $\theta$ for the purpose of the derivation. In the above case of the sample variance, the bias is easy to correct (just multiply the resulting estimate by $\frac{N}{N-1}$), but in general this may not be possible, e.g., if the bias depends on the true parameter vector $\theta$.

As a sidenote, in many cases it may be desirable to use a biased estimator because, e.g., the mean squared error of a biased estimator may be lower than the mean squared error of an unbiased estimator. See, e.g., the bias-variance trade-off. One classical example is the James-Stein estimator, which is biased but has a lower MSE than ordinary least squares, which is unbiased.

$\endgroup$
16
  • $\begingroup$ I think you must mean $\hat\sigma_Y^2.$ $\endgroup$
    – BruceET
    Aug 8 '21 at 16:39
  • $\begingroup$ Of course, thanks! Corrected. $\endgroup$
    – jhin
    Aug 8 '21 at 16:40
  • 2
    $\begingroup$ @Chuckha: The answers were fine but, in case this part of your question wasn't answered, the underlying distribution often does matter. For example, the mean is unbiased for the gaussian but it's not unbiased for say the chi-squared or the laplace. $\endgroup$
    – mlofton
    Aug 8 '21 at 16:54
  • 2
    $\begingroup$ Mlofton is wrong. When population mean is defined sample mean is unbiased for all distributions. It is just not efficient if distribution is not Gaussian. $\endgroup$ Aug 8 '21 at 18:00
  • 1
    $\begingroup$ I found a link which shows that, for the Laplace, E(X) = u. So the sample mean IS UNBIASED and doesn't even need asymptotics. So, my apologies to Cagdas and jhin for being wrong. I was definitely wrong. And also apologies for making this thread so confusing. It seems that the sample mean is always atleast asymptotically unbiased for the first parameter of it's distribution. News to me and thanks for teaching me something. I'll remove my rude comment to Cagdas but leave the rest in so as not confuse the thread even more. $\endgroup$
    – mlofton
    Aug 11 '21 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.