Confidence intervals when the sample size is very large

My question could be rephrased as "how to assess a sampling error using big data", especially for a journal publication. Here is an example to illustrate a challenge.

From a very large dataset (>100000 unique patients and their prescribed drugs from 100 hospitals), I interested in estimating a proportion of patients taking a specific drug. It's straightforward to get this proportion. Its confidence interval (e.g., parametric or bootstrap) is incredibly tight/narrow, because n is very large. While it's fortunate to have a large sample size, I'm still searching for a way to assess, present, and/or visualize some forms of error probabilities. While it seems unhelpful (if not misleading) to put/visualize a confidence interval (e.g., 95% CI: .65878 - .65881), it also seems impossible to avoid some statements about uncertainity.

Please let me know what you think. I would appreciate any literature on this topic; ways to avoid over-confidence in data even with a large sample size.

• You can avoid over-confidence by recalling that non-sampling errors remain untouched. If there are biases in sampling and measurement, they are still there. Also, whether you are counting unique (I'd rather say "distinct") patients or observations defined in some other way, there are (I presume) cluster structures linking drugs for the same patient and drugs that are given together any way, which are not accounted for by the simplest confidence interval calculations. I have no solutions on how to quantify this beyond comparing with other datasets and documenting data production. – Nick Cox Apr 7 '15 at 19:25

This problem has come up in some of my research as well (as a epidemic modeler, I have the luxury of making my own data sets, and with large enough computers, they can be essentially arbitrarily sized. A few thoughts:

• In terms of reporting, I think you can report more precise confidence intervals, though the utility of this is legitimately a little questionable. But it's not wrong, and with data sets of this size, I don't think there's much call to both demand confidence intervals be reported and then complain that we'd really all like them to be rounded to two digits, etc.
• In terms of avoiding overconfidence, I think the key is to remember that precision and accuracy are different things, and to avoid trying to conflate the two. It is very tempting, when you have a large sample, to get sucked into how very precise the estimated effect is and not think that it might also be wrong. That I think is the key - a biased data set will have that bias at N = 10, or 100, or 1000 or 100,000.

The whole purpose of large data sets is to provide precise estimates, so I don't think you need to shy away from that precision. But you do have to remember that you can't make bad data better simply by collecting larger volumes of bad data.

• I think a large volume of bad data is still better than small volume of bad data. – Aksakal Apr 7 '15 at 19:21
• @Aksakal Why? A precisely wrong answer is still wrong. – Fomite Apr 7 '15 at 21:10
• @Fomite - yeah, but you're more confident that it's wrong :) – Duncan Apr 7 '15 at 21:59

This problem has come up in my own manuscripts.

1. Reporting Options: If you have just one or a few CIs to report, then reporting "(e.g., 95% CI: .65878 - .65881)" is not overly verbose, and it highlights the precision of the CI. However, if you have numerous CIs, then a blanket statement might be more helpful to the reader. For example, I'll usually report something to the effect of "with this sample size, the 95% confidence margin of error for each proportion was less than +/- .010." I usually report something like this in the Method, or in the caption of Table or Figure, or in both.

2. Avoiding "over-confidence" even with large sample size: With a sample of 100,000, the central limit theorem will keep you safe when reporting CIs for proportions. So, in the situation you described, you should be okay, unless there are other assumption violations I'm not aware of (e.g., violated i.i.d.).

Don't report confidence intervals. Instead report the exact sample size and the proportions. The reader will be able to calculate his own CIs in any way he wishes.

• Why shouldn't this very reasoning be applied to all reporting of quantitative data? – whuber Apr 7 '15 at 19:40
• @whuber, good question. I'm all for reproducible research, wish everyone published their datasets. – Aksakal Apr 7 '15 at 19:46
• I didn't mean for it to be taken as a suggestion. Even if everyone published their datasets, they would be abrogating their scientific duties if they failed to supply an analysis of them--and that includes an analysis of uncertainty. You seem to be going in a direction that logically would end with the suggestion that scientists do nothing but publish data, with no analysis at all! That winds up being an indictment of the recommendation that CIs not be reported. It indicates to the contrary that some kind of statistical analysis should be offered in any case, regardless of sample size. – whuber Apr 7 '15 at 22:19

Consider the possibility that the 100 different hospitals' proportions do not converge to the same mean value. Did you test for between-group variance? If there is a measurable difference between hospitals, then the assumption that the samples are generated from a common normal distribution is not supported & you should not pool them.

However if your data really does come from a normally distributed large sample, then you are not going to find useful "statements about uncertainty" as a property of the data, but upon reflection about why or why not your statistics should generalize -- due to some inherent bias in collection, or lack of stationarity, etc. that you should point out.