I have a data set with two groups. I have multiple trials with different Ns in each trial, and different numbers of trials for each group. In each trial, for each group, I am counting the number of successes.

How can I estimate the proportion of successes for each group under a binomial distribution (i.e. success or failure), with confidence intervals? How can I test whether the proportion of successes is the same between the two groups?

The standard Z-test for proportions is only described for one trial per group.

This question wasn't sufficient: Determining statistical significance of difference between two binomial distributions

Nor was this: Test if two binomial distributions are statistically different from each other

  • $\begingroup$ The N samples within a single trial are not perfectly independent, but it is fairly reasonable to assume independence. $\endgroup$ – Dylan Richard Muir Jul 3 '15 at 9:22
  • $\begingroup$ Do you have the outcomes for each individual in each trial (in each group)? $\endgroup$ – Daniel Jul 3 '15 at 9:29
  • $\begingroup$ @Daniel: Sorry if I misunderstand your question. For each of the N individual samples in each trial (in each group), I know the outcome -- either success or failure. $\endgroup$ – Dylan Richard Muir Jul 3 '15 at 9:33
  • $\begingroup$ But do you know which individual is responsible for each sample in each trial? e.g. if group A has 3 individuals (a, b, c), and the first trial for group A has 5 samples, do you have (1, 0, 1, 1, 0), or ({1, b}, {0, a}, {1, c}, {1, a}, {0, b})? $\endgroup$ – Daniel Jul 3 '15 at 10:57
  • $\begingroup$ Individuals are only sampled once, and are not repeated across groups or across trials. Individual ID can therefore be neglected: group A trial 1 is (1, 0, 1, 1, 0). $\endgroup$ – Dylan Richard Muir Jul 3 '15 at 14:24

If I'm understanding your question right (and a couple simple assumptions are met), the second link you have gives you what you're looking for. So a simple probability fact is that if X and Y are independent, X~Bin(n,p) and Y~Bin(m,p) then X+Y~Bin(m+n,p). That means that if the trials in each of your groups are independent then

$\sum_{i=1}^{|G_{1}|}X_{i}$ ~Bin($\sum_{i=1}^{|G_{1}|}N_{i}^{1}$, p) and $\sum_{i=1}^{|G_{2}|}Y_{i}$ ~ Bin($\sum_{i=1}^{|G_{2}|}N_{i}^{2}$,p)

where the G's are the groups, the N_i are the n for each binomial trial (superscript tells the group) and |G| is the size of G. From there, you're back at the problem of estimating proportions and confidence intervals in binomial families. The big things to make sure of are that the trials within groups are independent and that you have no reason to believe the probabilities of success are different between trials within groups.

  • $\begingroup$ And what if the probability of success differs between trials within groups? For example, if there are differences in how difficult it is to detect "success" between trials? $\endgroup$ – Dylan Richard Muir Jul 13 '15 at 9:40
  • $\begingroup$ Using the same kind of logic as above (binomials as sums of bernoulli's) you can treat your sums as poisson binomial's. They're not particularly nice but at least you have a distribution for them. en.wikipedia.org/wiki/Poisson_binomial_distribution $\endgroup$ – Nick Thieme Jul 13 '15 at 14:12
  • $\begingroup$ Are there any tests that would operate on P.B. distributed sets of values? Or would one have to roll-your-own monte carlo test? $\endgroup$ – Dylan Richard Muir Jul 13 '15 at 15:01
  • 1
    $\begingroup$ Here's a link to a better explanation than I could ever give:stats.stackexchange.com/questions/93645/… $\endgroup$ – Nick Thieme Jul 13 '15 at 15:08

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.