Why does unbiasedness not imply consistency

Question

I'm reading deep learning by Ian Goodfellow et al. It introduces bias as $$Bias(\theta)=E(\hat\theta)-\theta$$ where $\hat\theta$ and $\theta$ are the estimated parameter and the underlying real parameter, respectively.

Consistency, on the other hand, is defined by $$\lim_{m\to\infty}\hat\theta_m=\theta$$ meaning that for any $\epsilon > 0$, $P(|\hat\theta_m-\theta|>\epsilon)\to0$ as $m\to\infty$

Then it says consistency implies unbiasedness but not vice versa:

Consistency ensures that the bias induced by the estimator diminishes as the number of data examples grows. However, the reverse is not true—asymptotic unbiasedness does not imply consistency. For example, consider estimating the mean parameter μ of a normal distribution N (x; μ, σ2 ), with a dataset consisting of m samples: ${x^{(1)}, . . . , x^{(m)}}$. We could use the first sample $x^{(1)}$ of the dataset as an unbiased estimator: $\hatθ = x^{(1)}$. In that case, $E(\hat θ_m) = θ$ so the estimator is unbiased no matter how many data points are seen. This, of course, implies that the estimate is asymptotically unbiased. However, this is not a consistent estimator as it is not the case that $\hatθ_m → θ$ as $m → ∞$

I'm not sure whether I've understood the above paragraph and the concepts of unbiasedness and consistency correctly, I hope someone could help me check it. Thanks in advance.

As far as I understand, consistency implies both unbiasedness and low variance and therefore, unbiasedness alone is not sufficient to imply consistency.

Answer 1

In that paragraph the authors are giving an extreme example to show how being unbiased doesn't mean that a random variable is converging on anything.

The authors are taking a random sample $X_1,\dots, X_n \sim \mathcal N(\mu,\sigma^2)$ and want to estimate $\mu$. Noting that $E(X_1) = \mu$, we could produce an unbiased estimator of $\mu$ by just ignoring all of our data except the first point $X_1$. But that's clearly a terrible idea, so unbiasedness alone is not a good criterion for evaluating an estimator. Somehow, as we get more data, we want our estimator to vary less and less from $\mu$, and that's exactly what consistency says: for any distance $\varepsilon$, the probability that $\hat \theta_n$ is more than $\varepsilon$ away from $\theta$ heads to $0$ as $n \to \infty$. And this can happen even if for any finite $n$ $\hat \theta$ is biased. An example of this is the variance estimator $\hat \sigma^2_n = \frac 1n \sum_{i=1}^n(y_i - \bar y_n)^2$ in a normal sample. This is biased but consistent.

Intuitively, a statistic is unbiased if it exactly equals the target quantity when averaged over all possible samples. But we know that the average of a bunch of things doesn't have to be anywhere near the things being averaged; this is just a fancier version of how the average of $0$ and $1$ is $1/2$, although neither $0$ nor $1$ are particularly close to $1/2$ (depending on how you measure "close").

Here's another example (although this is almost just the same example in disguise). Let $X_1 \sim \text{Bern}(\theta)$ and let $X_2 = X_3 = \dots = X_1$. Our estimator of $\theta$ will be $\hat \theta(X) = \bar X_n$. Note that $E \bar X_n = p$ so we do indeed have an unbiased estimator. But $\bar X_n = X_1 \in \{0,1\}$ so this estimator definitely isn't converging on anything close to $\theta \in (0,1)$, and for every $n$ we actually still have $\bar X_n \sim \text{Bern}(\theta)$.

Answer 2

As far as I understand, consistency implies both unbiasedness and low variance and therefore, unbiasedness alone is not sufficient to imply consistency.

Right. Or using the slightly more lay terms of "accuracy" for low bias, and "precision" for low variance, consistency requires that we be both accurate and precise. Just being accurate doesn't mean we're hitting the target. It's like the old joke about two statisticians who go hunting. One misses a deer ten feet to the left. The other one misses ten feet to the right. They then congratulate each other on the basis that, on average, they hit the deer. Even though their bias is zero, to actually hit the deer, they need low variance as well.