Why do we need an estimator to be consistent?

Question

I think, I have already understood the mathematical definition of a consistent estimator. Correct me if I'm wrong:

$W_n$ is an consistent estimator for $\theta$ if $\forall \epsilon>0$

$$\lim_{n\to\infty} P(|W_n - \theta|> \epsilon) = 0, \quad \forall\theta \in \Theta$$

Where, $\Theta$ is the Parametric Space.  But I want to understand the need for an estimator to be consistent. Why an estimator that is not consistent is bad? Could you give me some examples?

I accept simulations in R or python.

Answer 1

If the estimator is not consistent, it won't converge to the true value in probability. In other words, there is always a probability that your estimator and true value will have a difference, no matter how many data points you have. This is actually bad, because even if you collect immense amount of data, your estimate will always have a positive probability of being some $\epsilon>0$ different from the true value. Practically, you can consider this situation as if you're using an estimator of a quantity such that even surveying all the population, instead of a small sample of it, won't help you.

Answer 2

Consider $n = 10\,000$ observations from the standard Cauchy distribution, which is the same as Student's t distribution with 1 degree of freedom. The tails of this distribution are sufficiently heavy that it has no mean; the distribution is centered at its median $\eta = 0.$

A sequence of sample means $A_j = \frac 1j \sum_{i=1}^j X_i$ is not consistent for the center of the Cauchy distribution. Roughly speaking, the difficulty is that very extreme observations $X_i$ (positive or negative) occur with sufficient regularity that there is no chance for $A_j$ to converge to $\eta = 0.$ (The $A_j$ are not just slow to converge, they don't ever converge. The distribution of $A_j$ is again standard Cauchy [proof].)

By contrast, at any one step in a continuing sampling process, about half of the observations $X_i$ will lie on either side of $\eta,$ so that the sequence $H_j$ of sample medians does converge to $\eta.$

This lack of convergence of $A_j$ and convergence of $H_j$ is illustrated by the following simulation.

set.seed(2019)  # for reproducibility
n = 10000;  x = rt(n, 1);  j = 1:n
a = cumsum(x)/j
h = numeric(n)
for (i in 1:n) {
  h[i] = median(x[1:i])  } 
 par(mfrow=c(1, 2))
 plot(j, a, type="l", ylim=c(-5, 5), lwd=2,
    main="Trace of Sample Mean")
  abline(h=0, col="green2")
  k = j[abs(x)>1000] 
  abline(v=k, col="red", lty="dotted")
 plot(j, h, type="l", ylim=c(-5, 5), lwd=2,
     main="Trace of Sample Median")
  abline(h=0, col="green2") 
par(mfrow=c(1, 1))

Here is a list of steps at which $|X_i| > 1000.$ You can see the effect of some of these extreme observations on the running averages in the plot at left (at the vertical red dotted lines).

k = j[abs(x)>1000]
rbind(k, round(x[k]))
   [,1] [,2] [,3]  [,4] [,5]  [,6]   [,7]  [,8]
k   291  898 1293  1602 2547  5472   6079  9158
  -5440 2502 5421 -2231 1635 -2644 -10194 -3137

Consistency in important in estimation: In sampling from a Cauchy population, the sample mean of a sample of $n = 10\,000$ observations is no better for estimating the center $\eta$ than just one observation. By contrast, the consistent sample median converges to $\eta,$ so larger samples produce better estimates.

Answer 3

A really simple of example of why it's important to think of consistency, which I don't think gets enough attention, is that of an over-simplified model.

As a theoretical example, suppose you wanted to fit a linear regression model on some data, in which the true effects were actually non-linear. Then your predictions cannot be consistent for the true mean for all combinations of covariates, while a more flexible may be able to. In otherwords, the simplified model will have shortcomings which cannot be overcome by using more data.

Answer 4

@BruceET has already given an excellent technical answer, but I'd like to add a point about the interpretation of it all though.

One of the fundamental concepts in statistics is that as our sample size increases, we can reach more precise conclusions about our underlying distribution. You could think of it as the notion that taking lots of samples eliminates the random jitter in the data, so we get a better notion of the underlying structure.

Examples of theorems in this vein are plentiful, but the most well-known is the Law of Large Numbers, asserting that if we have a family of i.i.d. random variables $(X_i)_{i\in\mathbb{N}} \ $ and $\mathbb{E}[X_1] < \infty$, then $$\frac{1}{n} \sum_{k = 1}^n X_k \rightarrow \mathbb{E}[X] \ \ \ \text{a.s.}$$

Now, to require an estimator to be consistent is to demand that it also follows this rule: As its job is to estimate an unknown parameter, we would like it to converge to that parameter (read: estimate that parameter arbitrarily well) as our sample size tends to infinity.

The equation

$$\lim_{n\to\infty} P(|W_n - \theta|> \epsilon) = 0, \quad \forall\epsilon > 0\ \forall\theta \ \in \Theta$$

is nothing else but convergence in probability of the random variables $W_n$ towards $\theta$, meaning that in some sense, a larger sample will get us closer and closer to the true value.

Now, if you want, you can look at it conversely: If that condition were to fail, then even with infinite sample size, there would be a "corridor" with positive width $[\theta - \varepsilon, \theta + \varepsilon]$ around $\theta$ and a nonzero probability that even with arbitrarily large sample size, our estimator will fall outside that corridor. And that would obviously violate the aforementioned idea, so consistency is a very natural condition on estimators to desire and enforce.