I have three groups of data, each with a binomial distribution (i.e. each group has elements that are either success or failure). I do not have a predicted probability of success, but instead can only rely on the success rate of each as an approximation for the true success rate. I have only found this question, which is close but does not seem to exactly deal with the this scenario.
To simplify down the test, let's just say that I have 2 groups (3 can be extended from this base case).
- Group 1 trials: $n_1$ = 2455
- Group 2 trials: $n_2$ = 2730
- Group 1 success: $k_1$ = 1556
- Group 2 success: $k_2$ = 1671
I don't have an expected success probability, only what I know from the samples. So my implied success rate for the two groups is:
- Group 1 success rate: $p_1$ = 1556/2455 = 63.4%
- Group 2 success rate: $p_2$ = 1671/2730 = 61.2%
The success rate of each of the sample is fairly close. However my sample sizes are also quite large. If I check the CDF of the binomial distribution to see how different it is from the first (where I'm assuming the first is the null test) I get a very small probability that the second could be achieved.
1-BINOM.DIST(1556,2455,61.2%,TRUE) = 0.012
However, this does not take into account any variance of the first result, it just assumes the first result is the test probability.
Is there a better way to test if these two samples of data are actually statistically different from one another?