# Given a set of binary samples out of a given population, how to calculate the probability that the true ratio of one value is greater than Z

Say you have a bag of N marbles; some are black and some are white.

You draw x marbles from the bag, and y of them are white.

What is the probability that the true ratio of white marbles in the bag is greater than Z?

For example, there are 100 marbles in the bag. You draw 10, of which 4 are white. What is the probability that the ratio of white marbles in the bag is greater than 0.3 (ie 30 or more white marbles)?

Edit: actually I guess what I really need to know is this, but with replacement. So you replace each marble before drawing the next one. Although I would be curious about both cases, with and without replacement.

• FWIW this question is to help us evaluate website analytics data. – Nathan Stretch Sep 1 '16 at 0:31

I'm going to answer the question as asked, as an abstract math problem. If you need advice on solving a real-world data-analysis problem, you should post a new question with full details.

Since you want to make a probability statement about a population quantity (viz., the true number of marbles), a Bayesian approach is called for. Since we're sampling with replacement, we can treat the bag as having infinitely many marbles and merely concern ourselves with the probability of drawing white each time. Then we can state the problem like this: if $B_1$, $B_2$, … are IID Bernoulli conditional on the parameter $p$, and $Z$ is some constant in $[0, 1]$, then given $x$ observations of the $B_i$s, of which $y$ were equal to 1 (i.e., white), what is the posterior probability that $p > Z$?

The conjugate prior of the Bernoulli distribution is the beta distribution, which has two (hyper)parameters $α$ and $β$. You can encode any prior beliefs you have about $p$ by setting $α$ and $β$ appropriately. If you have no precise prior belief, the Jeffreys prior, which sets each to 1/2, is a good default. (N.B. I've made a simplifying assumption by allowing $p$ to vary continuously in [0, 1] when in reality it can only take on $N + 1$ distinct values.)

Given all this, the posterior distribution of $p$ is another beta distribution, with parameters $y + α$ and $x - y + β$. So calculating the posterior probability of $p > Z$ just means using the CDF of the beta distribution, which is the regularized incomplete beta function. An easy way to compute this is to run 1 - pbeta(Z, y + alpha, x - y + beta) in R. For your example with 10 draws 4 of which are white and $Z = .3$ and using the Jeffreys prior, this is 1 - pbeta(.3, 4 + 1/2, 10 - 4 + 1/2), which is about .762.

The case of sampling without replacement is similar, but using the hypergeometric distribution instead of the Bernoulli distribution, and the beta-binomial distribution instead of the beta distribution.

• Either I'm misunderstanding or there's a flaw of some sort in that formula. I just tried with x = 1000, y = 10 (so observed p is 0.01), but for basically any Z chosen short of 1, I get extremely high probability. ie: 1 - qbeta(0.9999, 10+0.5 , 1000 - 10 + 0.5) ~= 0.97 – Nathan Stretch Sep 1 '16 at 1:17
• @NathanStretch My mistake, the CDF is pbeta, not qbeta. – Kodiologist Sep 1 '16 at 1:59
• Gotcha, seems to work well now. After a bit of reading (stats.stackexchange.com/questions/47771/…) I think I understand the logic behind it, and so can set reasonable alpha and beta as well. (In short, in this example, you would want to set alpha/(alpha+beta) equal to your best a priori guess of the ratio of white marbles in the bag, and the larger alpha and beta, the more confident the guess.) – Nathan Stretch Sep 1 '16 at 4:30
• @NathanStretch That's right. A handy way to think of it is that you're treating $p$ as if you previously saw $α$ white marbles and $β$ black marbles in $α + β$ draws. – Kodiologist Sep 1 '16 at 5:23
• Since this allows for p to vary continuously, it seems to me that the estimates would be less accurate when the true p is very low. For instance, if the true 'p' is one white marble per 1000, and you take 2000 samples of which one is white, the result of the formula with Z=0.002 will either be dominated by the initial conditions, or if those are set very small, it will underestimate p. Am I understanding correctly? And if so, is there a guide for how large y would need to be in absolute terms to discount this? – Nathan Stretch Sep 8 '16 at 20:42