Bayesian interval estimation. Let $\theta$ be P(Success) and the prior distribution be $\theta \sim
\mathsf{Beta}(.5,.5).$ Then if you observe $x$ successes in $n$ trials, the likelihood is proportional to $\theta^x(1-\theta)^{n-x},$ so that the
posterior distribution of $\theta$ conditional on the data is
$\mathsf{Beta}(.5+x,\, .5+ n-x).$ Then a 95% Bayesian posterior
probability interval uses quantiles .025 and .975 of the posterior
beta distribution. (To get a one-sided interval with an upper bound, use
quantile 0.95.)
If you are willing to overlook the difference in interpretation between a
Bayesian posterior (or credible) interval and a frequentist confidence interval,
you might use the Bayesian interval based on the Jeffreys prior as a confidence interval. With $x = 3$ successes in $n = 100$ trials, the one sided 95% Jeffreys
interval would give the upper bound $0.068.$ The corresponding two sided 95% interval is
$(0.0085, 0.0779).$ [Computation in R.]
qbeta(.95, 3.5, 97.5)
[1] 0.06875679
qbeta(c(.025, .975), 3.5, 97.5)
[1] 0.008520283 0.077887569
Frequentist Confidence intervals. In a frequentist context with a normal approximation, the Wald 95% CI is
of the form $\hat \theta \pm 1.96 \sqrt{\frac{\hat \theta(1-\hat\theta)}{n}}.$
where $\hat \theta = x/n.$
If $\theta$ is far from $1/2$ and $n$ is relatively small, the Wald interval (originally proposed as an asymptotic interval) may not have the advertised
95% coverage probability: first, because the normal approximation may not be accurate; second, because the standard error may not be well approximated by using $\hat \theta$ instead of $\theta.$ In the cases $x = 0$ and $x =n,$ the Wald interval degenerates to a point. Agresti ("plus-four"), Wilson, and Clopper-Pearson intervals are substantially better, mitigating the second difficulty, but not the first.
For comparison, the two-sided 95% Agresti CI (formula here) based on 3 successes in 100 trials
is $(0.0070, 0.0892).$
n = 104; p = 5/104; pm = c(-1,1)
p + pm*1.96*sqrt(p*(1-p)/n)
[1] 0.006961131 0.089192715
True coverage probabilities of frequentist CIs vary as $\theta = P(\text{Success})$ varies. Because of the discreteness of the binomial distribution, the variation can be surprisingly large. For $n = 100,$ the actual coverage probability of the "95%" Wald interval is below 95% for 'most' values of $\theta.$ The performance is
sufficiently bad that its use is now deprecated.
Agresti 95% CIs generally have close to the promised coverage probability (and sometimes a little better
near $\theta = 0$ and $1.)$ While other intervals may be a bit more accurate, the Agresti style of CI is widely used because its formula is easy to remember and use.
When regarded as a frequentist CI, the Jeffries interval is reasonably accurate for most values of $\theta.$ (Regarded as a Bayesian posterior probability interval, the Jeffries interval always contains 95% probability according to the posterior distribution. So from a Bayesian point of view, if you believe the Prior and trust the integrity of the data, then you believe
the interval. Remember that in a Bayesian framework $\theta$ is a random variable, not a fixed, unknown value.)
