I have some observed binomial data as a toy example say I observe 15 successes out of 100 trials.
This could only have come from one of two distributions. I have observed data from these known two distributions. So in effect I want to test whether my observed sample is more likely to come from the same distribution as either:
sample 1: 6 successes out of 20 trials
sample 2: 7 successes out of 60 trials
I feel like I am missing a very simple solution that has slipped my mind ...
Let's start out with a related problem. We have a known probability $p$ and two samples $(n_1, x_1)$ and $(n_2, x_2)$. We can calculate the probability of observing $(x_1, x_2)$ given $(n_1, n_2, p)$ as follows:
$$p(x_1, x_2) = {n_1 \choose x_1}{n_2 \choose x_2}p^{x_1}(1-p)^{n_1-x_1}p^{x_2}(1-p)^{n_2-x_2}$$
which evidently simplifies to
$$p(x_1, x_2) = {n_1 \choose x_1}{n_2 \choose x_2}p^{x_1+x_2}(1-p)^{n_1+n_2-x_1-x_2}$$
If we assume $p$ is unknown, we can easily form a plug-in estimate of $p(x_1,x_2)$ by finding the maximum likelihood estimate $\hat{p} = (x_1+x_2)/(n_1+n_2)$ and substituting in the above. By the standard properties of MLEs, this implies that we are calculating the MLE of $p(x_1, x_2)$ as well.
Let's try that with your two samples, performing the calculations twice for the two combinations of your observed binomial data with each of the two samples. We'll label the observed binomial data $(100,15)$ with index 0.
$$p(x_0, x_1) = \dots = 0.00668$$ $$p(x_0, x_2) = \dots = 0.01472$$
Now, if we are going to do a real comparison, we also have to include an estimate of the probabilities of $x_1$ and $x_2$ in the above two equations so as to get estimates for the complete set $(x_1, x_2, x_3)$ - otherwise the comparison is not complete. These estimates can be easily calculated by plugging the MLEs of the two probabilities into the Binomial distribution:
$$p(x_1) = {20 \choose 6}\left(\frac{6}{20}\right)^6\left(\frac{14}{20}\right)^{14} = 0.1916$$ $$p(x_2) = {60 \choose 7}\left(\frac{7}{60}\right)^7\left(\frac{53}{60}\right)^{53} = 0.1585$$
Combining the various probabilities gives:
$$p(x_0,x_1,x_2) = p(x_0,x_1)p(x_2) = 0.00106$$ $$p(x_0,x_1,x_2) = p(x_0,x_2)p(x_1) = 0.00282$$
It appears somewhat more likely that sample 2 and the observed data are drawn from the same distribution as that sample 1 and the observed data are.
If we want to be Bayesian, we can actually calculate a posterior probability of the two hypotheses. Let's put a simple Uniform(0,1) prior probability on the various probability parameters for both hypotheses (it'll be outweighed by the data anyway, so worrying about exactly which diffuse prior to use is rather pointless in this case) and assign prior probabilities $p(H_1) = p(H_2) = 0.5$, where $H_1$ indicates the hypothesis that sample 1 and the observed data go together and similarly for $H_2$.
Our unnormalized posterior probability distribution for $p$ and $H_1$ is:
$$f(p, H_1|D) \propto {n_0 \choose x_0}{n_1 \choose x_1}p^{x_0+x_1}(1-p)^{n_0+n_1-x_0-x_1} {n_2 \choose x_2}p_2^{x_2}(1-p_2)^{n_2-x_2}p(H_1)$$
where we use $D$ as a shortcut for all the data. Integrating out $p$ and $p_2$ results in:
$$p(H_1|D) \propto {n_0 \choose x_0}{n_1 \choose x_1} {n_2 \choose x_2}\beta(x_0+x_1,n_0+n_1-x_0-x_1)\beta(x_2,n_2-x_2) p(H_1) = 0.000653*p(H_1)$$
and, similarly, $p(H_2|D) \propto 0.00202p(H_2)$. Normalizing in the obvious way results in:
$$p(H_1|D) = \frac{0.000653P(H_1)}{0.000653P(H_1)+0.00202P(H_2)} = 0.244$$
again, not surprisingly, indicating a mild preference for sample 2.
