4
$\begingroup$

I tried searching the site but nothing came up.

I have a simple situation for a business problem - We have a population of 50 million files. We want to review a sample and see whether a file contains or does not contain "x". So it's binary. We will calculate a proportion of yes:no.

We want to be able to quantify how accurate our estimate is of the true population. "We are 95.. 99% confident that the true proportion is "x/y" +/- x%. From my research, confidence intervals cannot be interpreted in such a way. Rather, confidence intervals are supposed to be interpreted as "in the long-run, if we conduct many experiments, we are 95% confident that the CIs calculated for those samples will contain the true population parameter". Do we just take an arbitrary "large enough" random sample, calculate the proportion and assume that is the true proportion? I came across the binomial credible interval that can be interpreted as the former, but I know nothing of Bayesian inference unfortunately.

$\endgroup$
11
  • 2
    $\begingroup$ Do a standard binomial CI for starters. Also, 50M files does sound a lot so yeah, not going to be a 10" grep, but maybe you can still brute-force it intelligently within an hour or two. Why do inference when we can know an exact answer? If the files have time-stamps it might be more relevant to even look at the proportion per month/week/day and compare. That would be more relevant variations (e.g. "on any given day we saw Y% of file having the pattern of question, with an IQR of [y_1,y_2]" a statement someone with minimal Stats background can follow). $\endgroup$
    – usεr11852
    Dec 21, 2021 at 16:14
  • $\begingroup$ (also how many files out of the 50M are examined? Realistically, unless our prior is really strong, after the first couple of thousand sampled files it will become mostly irrelevant; I would focus more on the idea that we don't bias our sample by accident more than anything else) $\endgroup$
    – usεr11852
    Dec 21, 2021 at 16:22
  • $\begingroup$ We don't know if each file contains "x" without manually going into the files. We want to be able to take a sample and estimate the proportion of "yes:no" for the entire population. $\endgroup$
    – DataNoob7
    Dec 21, 2021 at 16:22
  • $\begingroup$ Sample calcs online say we need to review 384 files, which seems off (probably due to the nature of the population being so large). I figure it will be a random sample of a couple of thousand. Also my knowledge of sample size calcs is when the DV is continuous, not binary. But the other issue is trying to provide some level of assurance that the proportion we observe for the sample is an estimate of the population within X%, x range, etc. For this, I understand confidence intervals are commonly misinterpreted to say this, so I need another method (if it exists). $\endgroup$
    – DataNoob7
    Dec 21, 2021 at 16:27
  • 1
    $\begingroup$ No worries, we are good! :) See Greg's answer below. "we do a random sample of 3000, is the assumption that the 50M population has that same sample proportion reasonable?" If it is a random sample yes. $\endgroup$
    – usεr11852
    Dec 21, 2021 at 16:59

5 Answers 5

7
$\begingroup$

This type of problem is well-suited to a confidence/credible interval. It is unclear to me why you think a confidence interval is unsuitable. You ought to be able to get a reasonable inference for the true proportion using a Wilson score interval. You can compute this confidence interval (for a finite or infinite population) using the CONF.prop function in the stat.extend package in R. Here is an example where I use some randomly generated mock data to get the interval (see O'Neill (2021) for details about the properties of the Wilson score interval):

#Set significance level and population size
#In this case the population size is large so it could be ignored if desired
N     <- 50000000
ALPHA <- 0.05

#Generate some (mock) data
p <- 0.03
n <- 3000
set.seed(1)
DATA <- sample(c(0,1), size = n, replace = TRUE, prob = c(1-p,p))

#Compute the confidence interval
stat.extend::CONF.prop(alpha = ALPHA, x = DATA, N = N)

        Confidence Interval (CI) 
 
95.00% CI for proportion for population of size 5e+07 
Interval uses 3000 binary data points from data DATA with sample proportion = 0.0303 

[0.0247716637647717, 0.0370961961781321]
$\endgroup$
9
  • $\begingroup$ Agreed. (+1). I like binomial::binom.confint more cause it has all the different types too. (Thus you can really see that with 2000+ samples pretty much any way you cut them the CIs are very similar.) $\endgroup$
    – usεr11852
    Dec 21, 2021 at 21:23
  • $\begingroup$ @Ben I am referring to the Frequentist definition of confidence interval. From my understanding, the Frequentist definition of a CI relates to the methodology - That is, if I conduct my CI methodology on 100...1000...100000 random samples, I can expect 95% of the CIs calculated will contain the true population parameter. What I cannot say is: "I am 95% certain that the CI I calculated for the sample contains the true population parameter." So, from my understanding as others have pointed out, I need to use a Bayesian approach to develop a credible interval (not confidence interval). $\endgroup$
    – DataNoob7
    Dec 21, 2021 at 22:21
  • $\begingroup$ @DataNoob7: (cause you said "others") My position from the start was "Do a standard binomial CI for starters." If anything I argued that the Bayesian posterior CI, given the absence of strong priors, will be dominated by the likelihood of the data (i.e. the "frequentist" part) so any difference will be immaterial. The general point that both me and Greg had was that you need to learn some Bayesian Inference - especially given you try delve into the interpretation more.) $\endgroup$
    – usεr11852
    Dec 21, 2021 at 22:29
  • $\begingroup$ Yes. My apologies for glossing over your opinion in broad strokes. If a frequentist approach is what I need to do, and makes more sense, then my issue lies in the interpretation of said CI as I noted in my comment above. If I am mistaken regarding the interpretation of the CI please let me know. This is also discussed here $\endgroup$
    – DataNoob7
    Dec 21, 2021 at 22:52
  • $\begingroup$ @DataNoob7, I have amended my answer in this thread to address your concern about interpreting a confidence interval. I also recommend you read my answer in the thread you linked to. $\endgroup$ Dec 22, 2021 at 0:45
5
$\begingroup$

The short and best advice is for you to learn Bayesian inference.

The linked paper makes some very good points and gives some good recommendations, but those are not the only possible recommendations. Rejecting any use of confidence intervals based on the paper is like "throwing out the baby with the bathwater". There are definite problems with how some confidence intervals are interpreted, and using the wrong confidence interval procedure can lead to meaningless intervals (but using the wrong Bayesian credible interval procedure can also lead to meaningless inference).

The common approximation methods for confidence intervals for a proportion do not work very well when the true proportion is near 0 or 1. My preferred interval in that case (and even when not near 0 or 1) is a Bayesian credible interval (or even full posterior), but you should understand what assumptions are being made and what the interval means, not just plug numbers into a formula.

You can investigate the various confidence/credible/compatibility interval procedures that work well for your likely situation and choose the one that works best (but understand what they are really saying, what assumptions are being made, and what their limitations are).

$\endgroup$
7
  • 1
    $\begingroup$ I agree Greg (+1) just my commentary is that given there is no prior here (or a weak one in any case from what OP says) we are gilding the lily. With a few thousand samples any non-really strong prior will be washed away from the likelihood and what we are left is a superficial conversation if a minor difference between confidence and credible intervals is meaningful or just arguing for arguments sake. $\endgroup$
    – usεr11852
    Dec 21, 2021 at 17:04
  • $\begingroup$ Thanks you both for the information. Obviously, I am very green when it comes to this. I am unsure where to even begin to calculate credible intervals, or which priors or posteriors to even use. I did stumble across this calculator that notes the distributions used, so maybe this is what I should use for this problem. $\endgroup$
    – DataNoob7
    Dec 21, 2021 at 17:49
  • $\begingroup$ Here is a related thread. $\endgroup$ Dec 21, 2021 at 18:18
  • $\begingroup$ @usεr11852, my concern is more that a canned "credible interval" routine will be used, then interpreted using frequentist definitions. Even using proper Bayesian definitions there can be some subtle but important things to think through. For example consider if the sample results in 0 cases of X, now compare a beta-binomial highest posterior density interval from a Beta(1,1) prior to an equal tail credible interval from a Beta(2,2) prior. The difference may be small, but there is a very important implication to inference in that difference. $\endgroup$
    – Greg Snow
    Dec 21, 2021 at 19:50
  • 1
    $\begingroup$ @usεr11852, yes, the 2 intervals will be pretty similar, but one will include 0% and the other one will not; one posterior distribution will have its mode at 0%, the other one will be 0 at 0%. One will have the observed percentage as the most likely parameter value, the other one will exclude the observed percentage as an impossible value for the parameter value. That is a huge impact from a small change in the prior. $\endgroup$
    – Greg Snow
    Dec 22, 2021 at 17:46
5
$\begingroup$

Confidence intervals can absolutely provide the inference you are interested in. A confidence interval is a set of plausible hypotheses in the parameter space given the observed data. The confidence level is based on the frequency probability of the experiment. This allows you to make the claim, "I am 95% confident that the CI computed from my sample covers the unknown fixed true population parameter." It's analogous to knowing the bias of a coin is 0.95 in favor of heads (95% of the time the coin lands heads) and the confidence this knowledge of the long-run proportion imbues regarding the outcome of a single flip. If asked how confident you are that the coin will land heads (or has already landed heads), you would say you are 95% confident based on its long-run performance.

Often times papers are published with examples that do not use all of the available information in the likelihood when constructing a confidence interval and this is presented as evidence against using a confidence procedure. This is analogous to using only a fraction of the information in the likelihood when constructing a posterior credible interval. Both the credible interval and the confidence interval reside in the parameter space. The choice between constructing a Bayesian credible interval or a frequentist confidence interval is a matter of what you want to measure using probability, the experimenter or the experiment.

The most precise solution to your problem is to construct a confidence interval for the population proportion by inverting the CDF of a binomial distribution. Here are some related threads that discuss frequentist confidence intervals for a Bernoulli proportion compared to Bayesian credible intervals when the unknown fixed true proportion is near 0 or 1 [1], [2].

$\endgroup$
9
  • 1
    $\begingroup$ OK, I'll soften my first sentence. $\endgroup$ Dec 21, 2021 at 23:28
  • $\begingroup$ Based on many threads I have read or "professionals" I work with, it seems confidence intervals do not allow you to say: "I am 95% confident that the true population parameter resides within the confidence interval I created for this sample", but rather "if I do this procedure enough times, 95% of the CI for those samples will contain the true population parameter". $\endgroup$
    – DataNoob7
    Dec 22, 2021 at 1:34
  • $\begingroup$ You're saying the former. From what I read, if I want to have the interpretation of the first sentence, I will need to construct a Bayesian credible interval. I must admit, your detailed answers (which I extremely appreciative of) in the threads you linked are over my head. I will have to revisit this idea once I become more informed/take additional coursework. $\endgroup$
    – DataNoob7
    Dec 22, 2021 at 1:34
  • $\begingroup$ A confidence interval is not created "for" a sample but "from" a sample. Including this minor correction, I am saying your former statement is a consequence of your latter statement. Both statements are correct under the frequentist paradigm. Please see the discussion regarding a biased coin in my answer above. Some people with ulterior motives will spread misinformation about frequentist inference claiming, "it doesn't give you what you want," or, "there is no confidence in confidence intervals." $\endgroup$ Dec 22, 2021 at 12:37
  • $\begingroup$ Be certain to talk with a wide range of professionals and read a wide range of papers. I recommend papers by Bradly Efron, Min-ge Xie, D.A.S. Fraser, and of course myself. Here is a paper in the context of clinical drug development highlighting that Bayesian belief is objectively viewed as confidence based on frequency probability of the experiment. $\endgroup$ Dec 22, 2021 at 12:45
0
$\begingroup$

We want to be able to quantify how accurate our estimate is of the true population. "We are 95.. 99% confident that the true proportion is "x/y" +/- x%. From my research, confidence intervals cannot be interpreted in such a way.

You are correct. Our confidence (or belief) is undefined unless we start with a Bayesian prior belief. In that case, we can update to a posterior belief based on the experiment. But the posterior will depend on the prior. For example, suppose our prior belief is the proportion is about 50%. And suppose our confidence interval is [30%, 40%]. We still might not believe the true parameter is in that low range. Instead we believe the experiment is a fluke.

The confidence-interval approach avoids making any claims about what we should believe post-hoc. However, it does give a useful guarantee: no matter what the true proportion is, with 95% probability (or 99.9%, etc), it will be in the confidence interval. In other words, if you make a claim "the true proportion is in this interval", that claim will only be wrong with probability 5% (or 0.1%, etc).

But when you see the actual interval resulting from your experiment, you might use prior knowledge to decide whether the 5% event (or 0.1%, etc) happened or not. That would be a Bayesian approach as I mentioned above.

$\endgroup$
0
$\begingroup$

I don’t think you should rely on the article “The fallacy of placing confidence in confidence intervals” as the last word on frequentist confidence intervals as the description is both limited and one-sided. Find a standard text first.

Let’s be specific to your problem. There is an unknown population proportion p. Suppose I construct a standard binomial 95% confidence interval for p then we know in advance that 95% of the random samples that could be drawn will produce CIs that contain p. We also know that the CI from a particular sample is either right or wrong. For example, suppose a random sample is drawn and the observed CI is 0.22 < p < 0.29. Now we know that P( the statement “0.22 < p <0.29” is correct) = 0 or 1. This is standard classical statistics. In such a situation, I would state 95% confidence that 0.22 < p <0.29 in the sense that this particular CI is the outcome of a process that produces correct CIs 95% of the time. Sure, I don’t know whether the observed CI is correct or not, but the 95% accuracy of the process that generated it is somewhat reassuring. Either the observed CI contains p or we have suffered a rare event [Note the past tense here.] In particular, my 95% confidence claim is neither a probability, nor a posterior probability, nor a belief. The genius of Neyman was to use a different word, confidence. Confidences should not be manipulated as if they are probabilities.

There is an important caveat regarding such confidence claims. Confidence would be undermined if it was known that the confidence interval procedure that I was using had poor conditional properties. In such circumstances, it is better to base confidence claims on appropriate conditional probabilities rather than on the unconditional probability. As far as I know, the standard binomial CI procedure does not have any dramatic shortcomings in this regard.

Regarding Bayesian credible intervals, suppose for some prior you were to generate a 95% credible interval, say 0.23 < p < 30, for example. That is, for your given prior, the posterior probability that 0.23 < p < 30 is 0.95. That form of outcome may seem better than a CI to you, but don’t forget that P( the statement “0.23 < p < 30” is correct) = 0 or 1, just like for confidence intervals.

Finally, if you are considering such large sample sizes, then perhaps you should think about raising the desired confidence level.

$\endgroup$
2
  • $\begingroup$ Thanks for the comment. Then it is similar to as @GeoffreyJohnson stated. It's the fact that because the procedure for which we created this CI from the sample produces CIs that contain the true population parameter 95% of the time is what allows us to say "we are 95% confident that the true population parameter is within (a<p<b)". We are not making a probabilistic statement, but rather providing a level of confidence or a best estimate of where the true population parameter lies based on the random sample we have drawn. Does this make sense? $\endgroup$
    – DataNoob7
    Dec 23, 2021 at 18:20
  • $\begingroup$ Yes, it is perfectly acceptable to say you are 95% confident that the parameter lies in the CI in that sense. $\endgroup$ Dec 23, 2021 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.