22
$\begingroup$

Consistency is obviously a natural and important property estimators, but are there situations where it may be better to use an inconsistent estimator rather than a consistent one?

More specifically, are there examples of an inconsistent estimator which outperforms a reasonable consistent estimator for all finite $n$ (with respect to some suitable loss function)?

$\endgroup$
  • 1
    $\begingroup$ There is an interesting tradeoff in performance between consistency of model selection and parameter consistency in estimation problems using the lasso and its (many!) variants. This is detailed, e.g., in Bühlmann and van der Geer's recent text. $\endgroup$ – cardinal Jun 25 '12 at 18:46
  • $\begingroup$ Wouldn't the argument in my, now deleted, answer still hold? Namely: in small samples it is better to have an unbiased estimator with low variance. Or can one show that an consistent estimator always has lower variance than any other unbiased estimator? $\endgroup$ – Bob Jansen Jun 25 '12 at 18:53
  • $\begingroup$ Perhaps, @Bootvis! Do you have an example of an inconsistent estimator with low MSE? $\endgroup$ – MånsT Jun 25 '12 at 18:56
  • 3
    $\begingroup$ @Bootvis: If you happen to look at the extensive comments on an answer to a recent question asking about consistency vs. unbiasedness, you will see that a consistent estimator can have arbitrarily wild behavior of both the variance and bias (even, simultaneously!). That should remove all doubt regarding your comment. $\endgroup$ – cardinal Jun 25 '12 at 18:58
  • $\begingroup$ I thought I had from one of two books but apparently I was wrong about that too! The example is nowhere to be found. @cardinal: Sounds interesting, will check it out $\endgroup$ – Bob Jansen Jun 25 '12 at 18:58
25
$\begingroup$

This answer describes a realistic problem where a natural consistent estimator is dominated (outperformed for all possible parameter values for all sample sizes) by an inconsistent estimator. It is motivated by the idea that consistency is best suited for quadratic losses, so using a loss departing strongly from that (such as an asymmetric loss) should render consistency almost useless in evaluating the performance of estimators.


Suppose your client wishes to estimate the mean of a variable (assumed to have a symmetric distribution) from an iid sample $(x_1, \ldots, x_n)$, but they are averse to either (a) underestimating it or (b) grossly overestimating it.

To see how this might work out, let us adopt a simple loss function, understanding that in practice the loss might differ from this one quantitatively (but not qualitatively). Choose units of measurement so that $1$ is the largest tolerable overestimate and set the loss of an estimate $t$ when the true mean is $\mu$ to equal $0$ whenever $\mu \le t\le \mu+1$ and equal to $1$ otherwise.

The calculations are particularly simple for a Normal family of distributions with mean $\mu$ and variance $\sigma^2 \gt 0$, for then the sample mean $\bar{x}=\frac{1}{n}\sum_i x_i$ has a Normal$(\mu, \sigma^2/n)$ distribution. The sample mean is a consistent estimator of $\mu$, as is well known (and obvious). Writing $\Phi$ for the standard normal CDF, the expected loss of the sample mean equals $1/2 + \Phi(-\sqrt{n}/\sigma)$: $1/2$ comes from the 50% chance that the sample mean will underestimate the true mean and $\Phi(-\sqrt{n}/\sigma)$ comes from the chance of overestimating the true mean by more than $1$.

Losses

The expected loss of $\bar{x}$ equals the blue area under this standard normal PDF. The red area gives the expected loss of the alternative estimator, below. They differ by replacing the solid blue area between $-\sqrt{n}/(2\sigma)$ and $0$ by the smaller solid red area between $\sqrt{n}/(2\sigma)$ and $\sqrt{n}/\sigma$. That difference grows as $n$ increases.

An alternative estimator given by $\bar{x}+1/2$ has an expected loss of $2\Phi(-\sqrt{n}/(2\sigma))$. The symmetry and unimodality of normal distributions imply its expected loss is always better than that of the sample mean. (This makes the sample mean inadmissible for this loss.) Indeed, the expected loss of the sample mean has a lower limit of $1/2$ whereas that of the alternative converges to $0$ as $n$ grows. However, the alternative clearly is inconsistent: as $n$ grows, it converges in probability to $\mu+1/2 \ne \mu$.

Loss functions

Blue dots show loss for $\bar{x}$ and red dots show loss for $\bar{x}+1/2$ as a function of sample size $n$.

$\endgroup$
  • 2
    $\begingroup$ (+1) Your comment "consistency is best suited for quadratic losses" interests me also but it's not blatantly obvious to me (and perhaps others) where that comes from. Clearly convergence in $L_2$ is best suited for quadratic losses and $L_2$ convergence implies convergence in probability but what is the motivation for this quote in the context of almost sure convergence a.k.a. "strong consistency"? $\endgroup$ – Macro Jun 25 '12 at 20:45
  • 5
    $\begingroup$ @Macro The thinking is somewhat indirect and not intended to be rigorous but I believe it is natural: quadratic loss implies minimizing variance which (via Chebyshev) leads to convergence in probability. Whence, a heuristic for finding a counterexample should focus on losses which are so far from quadratic that such manipulations are unsuccessful. $\endgroup$ – whuber Jun 25 '12 at 21:48
  • 1
    $\begingroup$ I don't understand the basis of your comment, @Michael: look at the last graphic. The expected loss for the consistent estimator decreases to $1/2$ while that of the inconsistent estimator decreases (exponentially) to $0$: it is thus exponentially better than the consistent one as $n$ grows large. $\endgroup$ – whuber Jun 25 '12 at 23:47
  • 3
    $\begingroup$ @Michael OK, thank you for explaining that. In this context, with a non-quadratic loss, an "advantage" is not expressed terms of bias. One might criticize this loss function, but I don't want to reject it outright: it models situations where, for instance, the data are measurements of an item manufactured to certain tolerances and it would be disastrous (as in Shuttle o-ring failure or business bankruptcy disastrous) for the true mean to fall outside those tolerances. $\endgroup$ – whuber Jun 26 '12 at 2:24
  • 1
    $\begingroup$ (+1) Great answer, @whuber! I particularly like that it doesn't feel too pathological - I can think of many situations where this type of loss would be applicable. $\endgroup$ – MånsT Jun 26 '12 at 7:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.