# Testing for differences in very small proportions

This is a somewhat typical a/b test setup. However the proportions that I'd like to test for differences are very small (< 1%). Consequently a 20% difference, which is meaningful, is 0.001. That's a small effect size!

A power test of the chi-squared test for differences in proportions (same sample size) using the pwr package in R yields

difference of proportion power calculation for binomial distribution
(arcsine transformation)

h = 0.001
n1 = 1e+05
n2 = 1e+05
sig.level = 0.05
power = 0.05574725
alternative = two.sided


So power grows very slowly. How can I get around this?

The group a proportion is lower than group b (which contains the treatment we're interested in). As expected the group b proportion is higher than group a. Say group bs proportion is 0.007.

Can I use an exact binomial test on group a with the alternative based on being greater than group bs proportion?

When I do this I get a p-value of 0.97, indicating that it's very unlikely that group a will increase in conversion to trump group bs current position.

I guess the question becomes one of power with this test? Any comments on the validity of this?