2
$\begingroup$

Given some estimator T for a parameter θ, by definition T is unbiased if its bias B(T) is 0. It is asymptotically unbiased if B(T) is not 0, but some value that tends to 0 as n goes to infinity. My question is: how are these properties different in practical terms? Unbiasedness by itself is only true for infinitely large n: given that B(T) = E(T) - θ, and the expected value E(T) is the mean of T's values over infinitely large n, it would seem that the two properties are saying the same thing. So, given they have the same variance, how do you distinguish between an estimator:

  1. whose sampling average tends towards θ for large n [E(T) = θ, unbiased]
  2. whose sampling average tends towards some value that tends towards θ for large n [E(T) = θ + f(n), where f(n) tends to 0 for large n, asymptotically unbiased]
$\endgroup$
4
  • 1
    $\begingroup$ Welcome to Cross Validated! Why do you say that unbiasedness only happens for infinitely large $n?$ $\endgroup$
    – Dave
    Sep 8, 2022 at 11:01
  • $\begingroup$ I'm fairly confident that relative efficiency is used to compare aympstotically unbiased estimators, i.e. the idea is they get closer to unbiased at sample sizes. I'm not 100% that it applies only to unbiased or asmpytotically unbiased, but pretty sure it's both. en.wikipedia.org/wiki/… $\endgroup$
    – Huy Pham
    Sep 8, 2022 at 11:04
  • $\begingroup$ Thanks. My understanding is that an estimator is unbiased if its expected value is equal to the true value of the parameter we're trying to estimate. But the expected value is the limit as n goes to infinity of the mean of the values of the estimator. So when talking about simple unbiasedness we're already in a scenario where n is infinite. Or am I missing something? $\endgroup$ Sep 8, 2022 at 11:56
  • 2
    $\begingroup$ Try the German Tank problem. The MLE is $\max(x_i)$ which is obviously biased: this can be lower than the true value but cannot be higher. As $n$ increases, this is asymptotically unbiased, but for given $n$ there are what would be usually seen as better estimators. So it does make a practical difference. $\endgroup$
    – Henry
    Sep 8, 2022 at 12:54

1 Answer 1

2
$\begingroup$

Let $X_1,\dots,X_n\overset{iid}{\sim}N(\mu, 1)$, so the sample size is $n$.

Define two estimators of $\mu$.

  1. $\hat\mu_1 = \bar X = \dfrac{1}{n}\sum_{i=1}^n X_i$

  2. $\hat\mu_2 = \bar X = \dfrac{1}{n} + \dfrac{1}{n}\sum_{i=1}^n X_i$

Let's calculate the expected value of each.

$$ \mathbb E\left[\hat\mu_1\right] = \mathbb E\left[\dfrac{1}{n}\sum_{i=1}^n X_i\right] \\=\dfrac{1}{n}\mathbb E\left[\sum_{i=1}^n X_i\right] \\=\dfrac{1}{n}\mathbb \sum_{i=1}^nE\left[ X_i\right] \\=\dfrac{1}{n}\mathbb \sum_{i=1}^n\mu \\=\dfrac{1}{n} n\mu \\=\mu $$

For any finite number of samples, including just one sample, $\bar X$, the usual sample mean, is unbiased.

$$ \mathbb E\left[\hat\mu_2\right] = \mathbb E\left[\dfrac{1}{n}+\dfrac{1}{n}\sum_{i=1}^n X_i\right] \\=\dfrac{1}{n} + \dfrac{1}{n}\mathbb E\left[\sum_{i=1}^n X_i\right] \\=\dfrac{1}{n} + \dfrac{1}{n}\mathbb \sum_{i=1}^nE\left[ X_i\right] \\=\dfrac{1}{n} + \dfrac{1}{n}\mathbb \sum_{i=1}^n\mu \\=\dfrac{1}{n} + \dfrac{1}{n} n\mu \\=\dfrac{1}{n} + \mu $$ There is no finite $n$ for which $\hat\mu_2$ is unbiased, since there's always some little positive number added to $\mu$. However, as $n\rightarrow\infty$, that little positive number gets smaller and smaller, vanishing in the limit. Consequently, $\hat\mu_2$ is asymptotically unbiased.

I think you are mixing up the sample size and the number of times samples are drawn. We can draw finite sample sizes over and over. For instance, I will apply the above estimators in a simulation that draws two observations $1000$ times.

KEY POINT: The sample size is two, not $1000$ or $2000$.

library(ggplot2)
set.seed(2022)
n <- 2
R <- 1000
mu <- 0

muhat1 <- muhat2 <- rep(NA, R)
for (i in 1:R){
  
  x <- rnorm(n, mu, 1)
  muhat1[i] <- mean(x)
  muhat2[i] <- mean(x) + 1/n
  
}
d1 <- data.frame(estimator = "muhat1", value = muhat1)
d2 <- data.frame(estimator = "muhat2", value = muhat2)
d <- rbind(d1, d2)
ggplot(d, aes(x = value, fill = estimator)) +
  geom_density(alpha = 0.3) +
  theme_bw()

enter image description here

Even with a sample size of two, the distribution of $\hat\mu_1 = \bar X$ is centered around the true $\mu=0$.

However, the sample size is two, not $1000$. We do $1000$ replications of the draw of two in order to create a distribution, not because the sample size is $1000$ or $2000$. The estimators each apply to samples with sample sizes of two.

$\endgroup$
2
  • 2
    $\begingroup$ "I think you are mixing up the sample size and the number of times samples are drawn" Yes! Thank you! I think that's exactly what I was getting wrong. So if I have this right: if for example I use the biased sample variance estimator (without Bessel's correction) with a sample size of 2, then even after an infinite number of trials the expected value is half the true variance of the population. What has to increase for the bias to disappear is the size of each individual sample, not the number of attempts. "n" refers to the former when talking about asymptotic unbiasedness. $\endgroup$ Sep 8, 2022 at 12:45
  • $\begingroup$ I do not follow what you’re saying about variance; perhaps post that as a separate question. $\endgroup$
    – Dave
    Sep 8, 2022 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.