1
$\begingroup$

My data is made up of 4 groups, each with a large and identical sample size of 20,000. The number of successes in each group is fairly small:

Group             A          B           C          D
Sample         20,000     20,000      20,000     20,000
Successes       90         120         110         200
Pr(Success)     0.5%       0.6%        0.6%        1.0%

I've done a simple chi-square test and found a significant difference (hope that was right!). Is it now appropriate to then do t-tests or are the proportions too small and I should do a series of further chi-squared tests? (I'm generally unsure about if/ when t-test is appropriate for very small. large proportions even if np and nq >20).

Thanks!

$\endgroup$
  • $\begingroup$ Just recalled that chi-square may not be most suitable for this data as samples are so large, is there a more suitable test to use? $\endgroup$ – Hatti Jan 9 '15 at 10:23
2
$\begingroup$
  • RE: suitable test. I would use the Poisson distribution to model this situation rather than the binomial, since n is large and p is small. The null hypothesis is that all 4 rate parameters are the same vs alternative that at least one differs from the rest.
  • Having done this test, and if I got a significant result, I could then study which groups are different. You could do this simply by quoting the rates with their confidence intervals.
  • I assume that your "t-test" are supposed to do pair-wise comparisons of the means in the presence of a significant result for the overall test. In the normal variate case, post-hoc comparisons are handled through things like Tukey's HSD or a Bonferroni adjustment. Long story short: you need to do your pair-wise comparisons with a (much) smaller p-value than you would normally use, to compensate for the fact that you are doing many tests.
  • I don't actually like post-hoc comparisons since it's not clear what the true p-values are,even with adjustments. Since you only do the tests conditionally on having obtained a significant result earlier on, the true p-value depends on a rather complicated distribution -- not the usual null hypothesis distribution.

To me, all the information your study has to offer can be expressed in the overall test of equality of the means, plus quoted means and confidence intervals. If a particular comparison interests you, you could also quote a confidence interval for the difference in the rates (or their ratio, since we're dealing with Poisson variates).

$\endgroup$
  • $\begingroup$ Thanks a lot, I hadn't thought of using the Poisson- so does this apply even though it's strictly a binomial distribution with 20,000 individuals and each either responding (success) or not responding? $\endgroup$ – Hatti Jan 11 '15 at 21:17
  • $\begingroup$ yes. you can prove that a binomial distribution tends towards a Poisson when p tends to zero and n tends to infinity while keeping their ratio constant. $\endgroup$ – Placidia Jan 12 '15 at 11:31
  • $\begingroup$ great thanks- is there any good reading around how to decide acceptable values of n and p for which Poisson may be appropriate? Sorry for all questions! $\endgroup$ – Hatti Jan 12 '15 at 20:02
  • $\begingroup$ I don't know any rules of thumb, but your numbers should qualify. $\endgroup$ – Placidia Jan 12 '15 at 20:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.