Basic Sampling - Provide confidence of estimate I tried searching the site but nothing came up.
I have a simple situation for a business problem - We have a population of 50 million files. We want to review a sample and see whether a file contains or does not contain "x". So it's binary. We will calculate a proportion of yes:no.
We want to be able to quantify how accurate our estimate is of the true population. "We are 95.. 99% confident that the true proportion is "x/y" +/- x%. From my research, confidence intervals cannot be interpreted in such a way. Rather, confidence intervals are supposed to be interpreted as "in the long-run, if we conduct many experiments, we are 95% confident that the CIs calculated for those samples will contain the true population parameter". Do we just take an arbitrary "large enough" random sample, calculate the proportion and assume that is the true proportion? I came across the binomial credible interval that can be interpreted as the former, but I know nothing of Bayesian inference unfortunately.
 A: This type of problem is well-suited to a confidence/credible interval.  It is unclear to me why you think a confidence interval is unsuitable.  You ought to be able to get a reasonable inference for the true proportion using a Wilson score interval.  You can compute this confidence interval (for a finite or infinite population) using the CONF.prop function in the stat.extend package in R.  Here is an example where I use some randomly generated mock data to get the interval:
#Set significance level and population size
#In this case the population size is large so it could be ignored if desired
N     <- 50000000
ALPHA <- 0.05

#Generate some (mock) data
p <- 0.03
n <- 3000
set.seed(1)
DATA <- sample(c(0,1), size = n, replace = TRUE, prob = c(1-p,p))

#Compute the confidence interval
stat.extend::CONF.prop(alpha = ALPHA, x = DATA, N = N)

        Confidence Interval (CI) 
 
95.00% CI for proportion for population of size 5e+07 
Interval uses 3000 binary data points from data DATA with sample proportion = 0.0303 

[0.0247716637647717, 0.0370961961781321]

A: The short and best advice is for you to learn Bayesian inference.
The linked paper makes some very good points and gives some good recommendations, but those are not the only possible recommendations.  Rejecting any use of confidence intervals based on the paper is like "throwing out the baby with the bathwater".  There are definite problems with how some confidence intervals are interpreted, and using the wrong confidence interval procedure can lead to meaningless intervals (but using the wrong Bayesian credible interval procedure can also lead to meaningless inference).
The common approximation methods for confidence intervals for a proportion do not work very well when the true proportion is near 0 or 1.  My preferred interval in that case (and even when not near 0 or 1) is a Bayesian credible interval (or even full posterior), but you should understand what assumptions are being made and what the interval means, not just plug numbers into a formula.
You can investigate the various confidence/credible/compatibility interval procedures that work well for your likely situation and choose the one that works best (but understand what they are really saying, what assumptions are being made, and what their limitations are).
A: Confidence intervals can absolutely provide the inference you are interested in.  A confidence interval is a set of plausible hypotheses in the parameter space given the observed data.  The confidence level is based on the frequency probability of the experiment.  This allows you to make the claim, "I am 95% confident that the CI computed from my sample covers the unknown fixed true population parameter."  It's analogous to knowing the bias of a coin is 0.95 in favor of heads (95% of the time the coin lands heads) and the confidence this knowledge of the long-run proportion imbues regarding the outcome of a single flip.  If asked how confident you are that the coin will land heads (or has already landed heads), you would say you are 95% confident based on its long-run performance.
Often times papers are published with examples that do not use all of the available information in the likelihood when constructing a confidence interval and this is presented as evidence against using a confidence procedure.  This is analogous to using only a fraction of the information in the likelihood when constructing a posterior credible interval.  Both the credible interval and the confidence interval reside in the parameter space.  The choice between constructing a Bayesian credible interval or a frequentist confidence interval is a matter of what you want to measure using probability, the experimenter or the experiment.
The most precise solution to your problem is to construct a confidence interval for the population proportion by inverting the CDF of a binomial distribution.  Here are some related threads that discuss frequentist confidence intervals for a Bernoulli proportion compared to Bayesian credible intervals when the unknown fixed true proportion is near 0 or 1 [1], [2].
A: 
We want to be able to quantify how accurate our estimate is of the true population. "We are 95.. 99% confident that the true proportion is "x/y" +/- x%. From my research, confidence intervals cannot be interpreted in such a way.

You are correct. Our confidence (or belief) is undefined unless we start with a Bayesian prior belief. In that case, we can update to a posterior belief based on the experiment. But the posterior will depend on the prior. For example, suppose our prior belief is the proportion is about 50%. And suppose our confidence interval is [30%, 40%]. We still might not believe the true parameter is in that low range. Instead we believe the experiment is a fluke.
The confidence-interval approach avoids making any claims about what we should believe post-hoc. However, it does give a useful guarantee: no matter what the true proportion is, with 95% probability (or 99.9%, etc), it will be in the confidence interval. In other words, if you make a claim "the true proportion is in this interval", that claim will only be wrong with probability 5% (or 0.1%, etc).
But when you see the actual interval resulting from your experiment, you might use prior knowledge to decide whether the 5% event (or 0.1%, etc) happened or not. That would be a Bayesian approach as I mentioned above.
A: I don’t think you should rely on the article “The fallacy of placing confidence in confidence intervals” as the last word on frequentist confidence intervals as the description is both limited and one-sided. Find a standard text first.
Let’s be specific to your problem. There is an unknown population proportion p. Suppose I construct a standard binomial 95% confidence interval for p then we know in advance that 95% of the random samples that could be drawn will produce CIs that contain p. We also know that the CI from a particular sample is either right or wrong. For example, suppose a random sample is drawn and the observed CI is 0.22 < p < 0.29. Now we know that P( the statement “0.22 < p <0.29” is correct) = 0 or 1. This is standard classical statistics. In such a situation, I would state 95% confidence that 0.22 < p <0.29 in the sense that this particular CI is the outcome of a process that produces correct CIs 95% of the time. Sure, I don’t know whether the observed CI is correct or not, but the 95% accuracy of the process that generated it is somewhat reassuring. Either the observed CI contains p or we have suffered a rare event [Note the past tense here.] In particular, my 95% confidence claim is neither a probability, nor a posterior probability, nor a belief. The genius of Neyman was to use a different word, confidence. Confidences should not be manipulated as if they are probabilities.
There is an important caveat regarding such confidence claims. Confidence would be undermined if it was known that the confidence interval procedure that I was using had poor conditional properties. In such circumstances, it is better to base confidence claims on appropriate conditional probabilities rather than on the unconditional probability. As far as I know, the standard binomial CI procedure does not have any dramatic shortcomings in this regard.
Regarding Bayesian credible intervals, suppose for some prior you were to generate a 95% credible interval, say 0.23 < p < 30, for example. That is, for your given prior, the posterior probability that 0.23 < p < 30 is 0.95. That form of outcome may seem better than a CI to you, but don’t forget that P( the statement “0.23 < p < 30” is correct) = 0 or 1, just like for confidence intervals.
Finally, if you are considering such large sample sizes, then perhaps you should think about raising the desired confidence level.
