# Questions on issues with using Frequentist and Bayesian approach for the same test

One quick stats question, if I use Binomial Cumulative Distribution Function to get a sample size n for desired confidence level and tolerable error. Then we pick a sample of sample size n and find k failures in the sample. Is it ok for us to use Jeffreys Interval instead of the exact method (clopper Pearson interval) for getting the upper limit of the estimated failure rate in the population?

My concern is that the sample size is obtained using Frequentist while the Jeffreys Interval is Bayesian approach. So can we still draw conclusion on the upper limit of the estimated failure rate with a conference level?

Bayesian interval estimation. Let $$\theta$$ be P(Success) and the prior distribution be $$\theta \sim \mathsf{Beta}(.5,.5).$$ Then if you observe $$x$$ successes in $$n$$ trials, the likelihood is proportional to $$\theta^x(1-\theta)^{n-x},$$ so that the posterior distribution of $$\theta$$ conditional on the data is $$\mathsf{Beta}(.5+x,\, .5+ n-x).$$ Then a 95% Bayesian posterior probability interval uses quantiles .025 and .975 of the posterior beta distribution. (To get a one-sided interval with an upper bound, use quantile 0.95.)

If you are willing to overlook the difference in interpretation between a Bayesian posterior (or credible) interval and a frequentist confidence interval, you might use the Bayesian interval based on the Jeffreys prior as a confidence interval. With $$x = 3$$ successes in $$n = 100$$ trials, the one sided 95% Jeffreys interval would give the upper bound $$0.068.$$ The corresponding two sided 95% interval is $$(0.0085, 0.0779).$$ [Computation in R.]

qbeta(.95, 3.5, 97.5)
[1] 0.06875679
qbeta(c(.025, .975), 3.5, 97.5)
[1] 0.008520283 0.077887569


Frequentist Confidence intervals. In a frequentist context with a normal approximation, the Wald 95% CI is of the form $$\hat \theta \pm 1.96 \sqrt{\frac{\hat \theta(1-\hat\theta)}{n}}.$$ where $$\hat \theta = x/n.$$

If $$\theta$$ is far from $$1/2$$ and $$n$$ is relatively small, the Wald interval (originally proposed as an asymptotic interval) may not have the advertised 95% coverage probability: first, because the normal approximation may not be accurate; second, because the standard error may not be well approximated by using $$\hat \theta$$ instead of $$\theta.$$ In the cases $$x = 0$$ and $$x =n,$$ the Wald interval degenerates to a point. Agresti ("plus-four"), Wilson, and Clopper-Pearson intervals are substantially better, mitigating the second difficulty, but not the first.

For comparison, the two-sided 95% Agresti CI (formula here) based on 3 successes in 100 trials is $$(0.0070, 0.0892).$$

n = 104;  p = 5/104; pm = c(-1,1)
p + pm*1.96*sqrt(p*(1-p)/n)
[1]  0.006961131 0.089192715


True coverage probabilities of frequentist CIs vary as $$\theta = P(\text{Success})$$ varies. Because of the discreteness of the binomial distribution, the variation can be surprisingly large. For $$n = 100,$$ the actual coverage probability of the "95%" Wald interval is below 95% for 'most' values of $$\theta.$$ The performance is sufficiently bad that its use is now deprecated.

Agresti 95% CIs generally have close to the promised coverage probability (and sometimes a little better near $$\theta = 0$$ and $$1.)$$ While other intervals may be a bit more accurate, the Agresti style of CI is widely used because its formula is easy to remember and use.

When regarded as a frequentist CI, the Jeffries interval is reasonably accurate for most values of $$\theta.$$ (Regarded as a Bayesian posterior probability interval, the Jeffries interval always contains 95% probability according to the posterior distribution. So from a Bayesian point of view, if you believe the Prior and trust the integrity of the data, then you believe the interval. Remember that in a Bayesian framework $$\theta$$ is a random variable, not a fixed, unknown value.)

• thanks! we are using clopper-pearson interval for some conservative analysis because it is an exact method and will guarantee the coverage probability greater the nominal confidence level. Another question I have is when it is appropriate to use clopper-pearson? It is said that when p is super small or big, we should use clopper-pearson method. What will be the best way to prove the statement? Commented Dec 4, 2018 at 1:21
• Using a frequentist approach when $p$ is very near 0 (or 1), it seems best to use C-P because it guarantees selected coverage level. But near 0, 1, you may want a 1-sided CI to bound the probability away from 0 or 1. You should review proof of C-P to verify claim // In a practical Bayesian situation you might use an informative prior, and then the posterior distribution will give a one-or two-sided interval as desired. As I mentioned in my Answer, as long as you believe the prior and the data, then believing the posterior dist'n is a logical deduction and coverage probability is not an issue. Commented Dec 4, 2018 at 7:18