5
$\begingroup$

How would you demonstrate to a non-statistical audience (pictorially) that the consistency of an estimator matters?

The idea is the following. I have proved that a multivariate estimator people are interested in using is inconsistent. Unfortunately, this argument is not good enough since the community comprises primarily of doctors and health professionals who may not be well versed with this concept.

Any suggestions on graphical expositions people have found useful to drive home this point?

$\endgroup$
5
$\begingroup$

This is an indirect approach that might help lead you toward considering the question in a different light.

Let me play devil's advocate for a moment.

In practice*, how much does consistency matter?

* (you might think about whether your lay audience would care about anything else)

When you have data, you have some particular sample size, $n=n_0$. Certainly you care about behavior at that sample size. If you're pondering several possible sample sizes, behavior at those several sample sizes would matter.

I'm never likely to see a sample size of a trillion. But is consistency actually relevant even at a specific sample size of much larger order, like $n=10^{120}$? It doesn't tell me anything about the behavior at my actual sample size.

Why would behavior at the limit of some sequence of sample sizes that you will never see be of any consequence? There are certainly times when it might be convenient in some sense, or nice to have, but that alone isn't much of an argument that it's actually important.

If you can answer that question, you might see a way to motivate it to a lay audience.

If you have difficulty with that question, explaining it to a lay audience is not your first problem (your first problem would be more like why is it even important to you?).

$\endgroup$
4
$\begingroup$

Create a simple but realistic example, with a known ‘feature of the population’ (i.e., parameter). Simulate from this example, and create a plot, with the number of observations on the x-axis and ‘cumulative’ parameter estimates on the y-axis. Mark the population parameter with a red horizontal line. Point out that the estimates converges¹, but to a completely different value than the ‘real‘ value (i.e., the parameter), even if the number of observations are in the millions.

¹ If the estimator doesn’t converge, simply point this out. It may also be useful to show several realisations, to drive home the point.

$\endgroup$
3
$\begingroup$

Reasons to be Consistent, Part III:

1) Defense: smiling, an estimator property is "asymptotic" when we don't have a clue as to when it will actually start to visibly affect the behavior and the results of an estimator. It may take a sample of immense size, it may take a few dozens of observations. So we want to have consistency in order to safeguard against being led ashtray, and without even knowing it. Since patients' health is at stake, I guess this alone should be an argument that health professionals would listen to.

A graphical exposition could posses two values, the true value (and the associated treatment of the patient), and the probability limit of the estimator (wrong value), and its associated treatment. If the treatment in the second case is different (and possibly irrelevant/detrimental) than the one under consistency, then you have the potential danger the users will bear (and impose on the patient) by using the inconsistent estimator.

2) In most cases, inconsistency also means the existence of bias even if large amounts of information gather (although strictly speaking, the concept of (un)biasedness at a limiting situation has more than one definitions). So you could fall back to something like "even if sample sizes are small and you don't think that inconsistency matters, as measurements accumulate, if you pool them, their average will also be wrong" -since the averaging operation is something that everybody feels familiar with. So inconsistency makes pooling of the obtained estimates, or of the data proper, misleading, something that sabotages any mid-term / long-term attempt to uncover the true situation.

Reasons to be Helpful, Part III:
Can you give them a positive result? Is there an alternative estimator that performs the same job, and is also consistent? And if yes, how it compares as regards finite-sample properties, like bias, variance, Mean Squared Error?

Reasons to Worry, Part III: The real tough situation would be if a) there is no alternative or b) the alternative is consistent but it performs worse in finite sample properties. Here you enter into Risk and Decision Theory proper, in which case, @whuber should jump in and clear the fog.

$\endgroup$
2
$\begingroup$

Thank you so much for your responses.

I have had a lot of people in the discipline ask me "Who cares what happens at infinity? We are never going to get there"..similar to what Glen posted and my response has been "Why don't you consider the first entry in your sample as the mean of the sample?" though this seems a rather indirect answer to the question.

As a partial response to Glen, people in the medical field extensively use the bootstrap which critically depends on consistency. Alecos' answer hits a few other issues bang on!

My answer is in the part (b) reason to Worry domain of Alecos' answer. The answer obtained at present is something a lot of people want to believe so an alternative may not be well received.

Defending consistency turns out to be very difficult on a practical data set. I thought I could rely on the several fantastic texts on decision theory but all of them seem to pass inconsistency off as in inconvenience more so than a problem you need to be rid of.

To be more concrete, most texts in Theoretical Stats refrain from statements of the form "Inconsistent estimators are bad..." as opposed to ML texts which pretty much flat out state that "Overfitting is bad...". Can someone please explain why this is so?

$\endgroup$
  • $\begingroup$ a) The response to "why I don't just take the first observation in my sample" is already covered in my original answer (certainly I'd consider the properties at a sequence of sample sizes, if I am contemplating more than one potential sample size; that has nothing to do with consistency). I also didn't argue that I don't care about bias or efficiency, but I'll care about them at the sample size I have, not a sample size I'll never have. b) If I have a sample of say $50$, in what way do the properties of a bootstrap (bias, coverage, etc) on my data (i.e. at $n=50$) rely on consistency? $\endgroup$ – Glen_b Mar 15 '15 at 6:59
  • $\begingroup$ When you say you are considering properties at a sequence of sample sizes, you are implicitly acknowledging consistency. The ration behind your consideration is that as the sample size grows, I gather more information and therefore my variance is going to zer. As far as the bootstrap argument goes, see theorem 12.1 in stat.tamu.edu/~suhasini/teaching613/bootstrap.pdf especially the "Noting that" portion in brackets. The noting that portion implies $\bar{X}$ is a consistent estimator of $\mu$. You need to begin with a consistent estimator for your bootstrap samples to make sense $\endgroup$ – Sid Mar 15 '15 at 8:51
  • $\begingroup$ You have completely mistaken my meaning. I was responding to the "why not take the first observation of $n$". Let's say $n$ was 10. You set up a situation where I was considering whether to take 1 observation or 10. In that case, I say "I'm happy to compare 1 observation vs 10 for some estimator, and choose which I use. That has nothing to do with consistency, which is about $n\to\infty$, not $n=1$ vs $n=10$. Until I look at the estimator we're using and the conditions we assume, I don't know whether n=10 is better than n=1, but it's a comparison that can be made. $\endgroup$ – Glen_b Mar 15 '15 at 9:35

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.