In what settings would confidence intervals not get better as sample size increases?

Question

In a blog post, I have found the claim that

"I believe WG Cochrane the first point out (roughly 1970′s) that with confidence intervals in an observational setting, small sample sizes result in better coverage with large enough samples providing near zero coverage!"

Now I assume that the CI width should approach 0 with increasing sample size, but the idea that coverage would concurrently worsen is not convincing to me. Is this claim true, and under which circumstances? Or am I misreading it?

I've run a simulation using random normally distributed data with sample sizes from 10000 to 1000000 (one-sample t-test, 95% CI), 1000 runs at every sample size, and coverage did not get any worse for the higher sample sizes (instead, I found the expected near-constant ~5% error rate).

Answer 1

Note the qualification "in an observational setting".

Checking the context from which you've taken the quote (the subthread of comments that it is in), it looks like the intent is "in the real world" rather than in simulations, and probably doesn't include a controlled experiment ... and in that case, the likely intent is a consequence of the fact that the assumptions under which the intervals are derived don't actually quite hold. There are numerous things that can impact bias - which are of small effect compared to variability in small samples - but which generally don't reduce in size as sample size increases, while the standard errors do.

Since our calculations don't incorporate the bias, as intervals shrink (as $1/\sqrt n$), any unchanging bias, even if it's pretty small looms larger, leaving our intervals less and less likely to include the true value.

Here's an illustration - one which perhaps exaggerates bias - to indicate what I think is meant about CI coverage probability shrinking as sample size increases:

(Noting here that 'center of interval' refers to the center of the population of intervals, not the middle of any given sample's interval; the distance between the black and the red vertical lines is the bias, which is here being taken to be the same irrespective of sample size.)

Of course in any particular sample, the interval will be random - it will be wider or narrower and shifted left or right relative to the diagram, so that at any sample size it has some coverage probability between 0 and 1, but any amount of bias will make it shrink toward zero as $n$ increases. Here's an example with 100 confidence intervals at each sample size using simulated data (plotted with transparency, so the color is more solid where more intervals cover it):

Answer 2

Sweet irony. Before that paragraph, the same person says "No wonder there is such widespread confusion". "Confidence intervals in an observational setting": what does that even mean?

It appears to me that this is once again a confusion between estimation and hypothesis testing.

Now I know the CI width should approach 1 with increasing sample size.

No, it depends on the context. In principle, the width should converge to $0$. The coverage should be close to the nominal value for a large number of Monte Carlo simulations. The coverage does not depend on the sample size, unless some of the assumptions under which the CI was constructed are flawed (which maybe is what the OP meant to imply. "All the models are wrong", yeah.).

The reference is a comment in a post of a personal blog. I would not worry too much about the validity of this sort of reference. The blog, owned by Larry Wasserman, tends to be very well written on the other hand. This reminded me of the xkcd comic:

http://xkcd.com/386/