I have some observed binomial data as a toy example say I observe 15 successes out of 100 trials.

This could only have come from one of two distributions. I have observed data from these known two distributions. So in effect I want to test whether my observed sample is more likely to come from the same distribution as either:

sample 1: 6 successes out of 20 trials

sample 2: 7 successes out of 60 trials

I feel like I am missing a very simple solution that has slipped my mind ...

  • $\begingroup$ 1) Please add the self-study tag to self-study or homework questions. 2) What are the differences between the three observed probabilities (for a starting hint)? $\endgroup$
    – jbowman
    Dec 20, 2017 at 14:45
  • 1
    $\begingroup$ (1) this is not a homework question, just an adapted question from research (2) If you look at the differences between the observed probabilities how do you take into account the uncertainty of your estimates based on the number of trials? Not sure I understand what you mean... $\endgroup$
    – queenjobo
    Dec 20, 2017 at 17:08
  • $\begingroup$ It was a hint! Typically with stuff that is self-study, we try to guide you to the answer, rather than just providing one, That can result in long comment threads, though. Given that it's not homework, it's up to you as to how you'd like to proceed. I'm sure plenty of other people, including me, would be willing to work out an answer + explanation for you! $\endgroup$
    – jbowman
    Dec 20, 2017 at 17:15
  • $\begingroup$ Ah right ok, I was just trying to write the question down in a way that was the most understandable! Currently my best approach is doing two separate binomial proportion tests and comparing the two p-values but I would prefer a single test as I feel this is a bit hacky! Any help will be greatly appreaciated :) $\endgroup$
    – queenjobo
    Dec 20, 2017 at 17:23
  • $\begingroup$ Start by writing an expression for the likelihood of the data as a whole under each of the two models - one in which the 15 out of 100 share a common Bernoulli parameter with Sample 1, & one in which they share a common Bernoulli parameter with Sample 2. Then decide whether you want to (1) simply compare likelihoods (maximized over other unknown parameters) or 2) perform a hypothesis test with one of the models defining a null hypothesis (note the models are non-nested), or (3) carry out a Bayesian calculation of posterior probabilities. $\endgroup$ Dec 20, 2017 at 17:48

1 Answer 1


Let's start out with a related problem. We have a known probability $p$ and two samples $(n_1, x_1)$ and $(n_2, x_2)$. We can calculate the probability of observing $(x_1, x_2)$ given $(n_1, n_2, p)$ as follows:

$$p(x_1, x_2) = {n_1 \choose x_1}{n_2 \choose x_2}p^{x_1}(1-p)^{n_1-x_1}p^{x_2}(1-p)^{n_2-x_2}$$

which evidently simplifies to

$$p(x_1, x_2) = {n_1 \choose x_1}{n_2 \choose x_2}p^{x_1+x_2}(1-p)^{n_1+n_2-x_1-x_2}$$

If we assume $p$ is unknown, we can easily form a plug-in estimate of $p(x_1,x_2)$ by finding the maximum likelihood estimate $\hat{p} = (x_1+x_2)/(n_1+n_2)$ and substituting in the above. By the standard properties of MLEs, this implies that we are calculating the MLE of $p(x_1, x_2)$ as well.

Let's try that with your two samples, performing the calculations twice for the two combinations of your observed binomial data with each of the two samples. We'll label the observed binomial data $(100,15)$ with index 0.

$$p(x_0, x_1) = \dots = 0.00668$$ $$p(x_0, x_2) = \dots = 0.01472$$

Now, if we are going to do a real comparison, we also have to include an estimate of the probabilities of $x_1$ and $x_2$ in the above two equations so as to get estimates for the complete set $(x_1, x_2, x_3)$ - otherwise the comparison is not complete. These estimates can be easily calculated by plugging the MLEs of the two probabilities into the Binomial distribution:

$$p(x_1) = {20 \choose 6}\left(\frac{6}{20}\right)^6\left(\frac{14}{20}\right)^{14} = 0.1916$$ $$p(x_2) = {60 \choose 7}\left(\frac{7}{60}\right)^7\left(\frac{53}{60}\right)^{53} = 0.1585$$

Combining the various probabilities gives:

$$p(x_0,x_1,x_2) = p(x_0,x_1)p(x_2) = 0.00106$$ $$p(x_0,x_1,x_2) = p(x_0,x_2)p(x_1) = 0.00282$$

It appears somewhat more likely that sample 2 and the observed data are drawn from the same distribution as that sample 1 and the observed data are.

If we want to be Bayesian, we can actually calculate a posterior probability of the two hypotheses. Let's put a simple Uniform(0,1) prior probability on the various probability parameters for both hypotheses (it'll be outweighed by the data anyway, so worrying about exactly which diffuse prior to use is rather pointless in this case) and assign prior probabilities $p(H_1) = p(H_2) = 0.5$, where $H_1$ indicates the hypothesis that sample 1 and the observed data go together and similarly for $H_2$.

Our unnormalized posterior probability distribution for $p$ and $H_1$ is:

$$f(p, H_1|D) \propto {n_0 \choose x_0}{n_1 \choose x_1}p^{x_0+x_1}(1-p)^{n_0+n_1-x_0-x_1} {n_2 \choose x_2}p_2^{x_2}(1-p_2)^{n_2-x_2}p(H_1)$$

where we use $D$ as a shortcut for all the data. Integrating out $p$ and $p_2$ results in:

$$p(H_1|D) \propto {n_0 \choose x_0}{n_1 \choose x_1} {n_2 \choose x_2}\beta(x_0+x_1,n_0+n_1-x_0-x_1)\beta(x_2,n_2-x_2) p(H_1) = 0.000653*p(H_1)$$

and, similarly, $p(H_2|D) \propto 0.00202p(H_2)$. Normalizing in the obvious way results in:

$$p(H_1|D) = \frac{0.000653P(H_1)}{0.000653P(H_1)+0.00202P(H_2)} = 0.244$$

again, not surprisingly, indicating a mild preference for sample 2.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.