Bayes' Theorem and Agresti-Coull: Will it blend?

Question

I'd like to use Bayes' Theorem on data obtained through a small random sample, and I want to use Agresti-Coull (or any other alternative technique) to know how big the uncertainty is.

Here is Bayes' Theorem:

$P(A|B) = \frac{P(B|A)\cdot P(A)}{P(B)}$

Now, all the data I have on this system is obtained from small random samples, so there's a large uncertainty involved with all three variables, $P(B|A)$, $P(A)$ and $P(B)$.

I've been using Agresti-Coull to obtain both the value and the uncertainty for each of these three variables. (I represent the number+-uncertainty as a ufloat object using the uncertainties package.)

But using Agresti-Coull three times separately for these three variables is a problem; They are dependent on each other. So I've been getting impossible results. For example, if you let $P(B)$'s uncertainty pull it downward, and the respective uncertainties of $P(B|A)$ and $P(A)$ pull them upwards, you get a total probability bigger than one.

Is there a way to do Agresti-Coull-style approximation on the whole Bayes expression instead of doing it on the three pieces separately?

Answer 1

When applying the formula for P(B|A) for Agresti-Coull, it seems important to me to use, for the denominator (ñ), a number with uncertainty. The formula ñ=P(A)*N+4 (where N is the size of your sample) gives you this number, after you calculate P(A) with an uncertainty. With the uncertainties package, this would be:

# Calculation of P(A):
P_A = ufloat(…, …)
# Calculation of P(B|A):
P_B_A = …/(P_A*N+4)

Thus, P(B|A) is automatically correlated to P(A).

Furthermore, you must make sure that you feed ufloat() with standard deviations. This mean using a particular z_{1-alpha/2} value, in the Agresti-Coull formula.

Hope this helps!

Answer 2

Error propagation won't work in the way handled by the uncertainties package. As you note, they're dependent, so you have to take the covariances into account.

You can obtain the variance of your distribution P(B|A) using the Delta Method and use that to obtain a confidence interval.

With Bayesian inference, you might find it simpler to use a credible interval. The following slides do a good job of explaining how to obtain this:

• Bayesian analysis of one, two, and n-parameter models
• A Brief Tutorial on Bayesian Thinking

Answer 3

Brown, Cai, and DasGupta, AS, 2002

Brown, Cai, and DasGupta, Stat Sci, 2001

I don't know if I understand you correctly, but in my knowledge the above two papers are the most cited ones recently when it comes to binomial proportions' CI and estimation.

Sorry if this is not what you wanted.

Answer 4

This is a potential answer to the title of the original question and not necessarily the body of the question...

Looking at the Agresti confidence interval measurement, to my eyes it bears a resemblance to a Bayesian estimator.

https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Agresti-Coull_Interval

Looking at p = 1/n * ( X + 1/2 * z^2), I suspect the 1/2 could be considered Bayesian prior knowledge.

Let’s say for example, that one conducts a congressional approval poll and historically those polls have an approval rating of 0.35. Using a Bayesian train of thought perhaps one could use that information as prior knowledge and feed it in by using 0.35 instead of the 1/2 coefficient?

I suspect one could do so successfully.