Significance test for highly skewed Bernoulli distribution I am working with two highly skewed Bernoulli distributions where 96-99+% of the samples are in the "false" category, and the rest are in the "true" category (sort of speak). I am looking for a two-sided test of difference of proportions between the two samples. I can often achieve 500+ "trues" and tens or hundreds of thousands of "falses" in a reasonable time but I'm not sure if approximation to the normal distribution can withstand this extreme skewness.
I initially thought I might need something non-parametric, but here, I actually know the distribution.
I have been using a student's t-test, while paying attention to sample size estimation, but past experience has led me to be skeptical of its results. Thanks for your help.
 A: One common rule of thumb is that you can safely use the normal approximation when comparing two proportions as long as there are at least 10 "Trues" and 10 "Falses" in each group, if you have 500+ trues in each group then that is greater than 10 and using the normal approximation is still reasonable.  You can convince yourself of this by simulating several datasets with the sample sizes and proportions that you have, then computing the test statistic for the normal approximation (simulate under the null of equal proportions), then plot a histogram of all these simulated test statistics.  If this histogram is approximately normal then a normal approximation is fine.
You could also use Fisher's exact test (probably use sampling to get the estimated p-value, your sample size will require quite a bit of time to compute the exact p-value from this test).
A: Here's the probability function and distribution of the proportion of True (plus the normal approximation at which the chi-square will be exact) in a sample of size 10,000 and a proportion of True of only 1% (right below the low end of your suggested total sample size, and with expected number of True only one fifth of your suggested minimum):

Don't forget that you'll have about five times the expected successes shown here; your approximation will be much better than this.
A straight two sample proportions test or a chi-square test should do just fine. Indeed, one tenth of your proportion of True's would be just fine. One hundredth, you'd just go to exact methods.
