This is a somewhat typical a/b test setup. However the proportions that I'd like to test for differences are very small (< 1%). Consequently a 20% difference, which is meaningful, is 0.001. That's a small effect size!
A power test of the chi-squared test for differences in proportions (same sample size) using the
pwr package in
difference of proportion power calculation for binomial distribution (arcsine transformation) h = 0.001 n1 = 1e+05 n2 = 1e+05 sig.level = 0.05 power = 0.05574725 alternative = two.sided
So power grows very slowly. How can I get around this?
a proportion is lower than group
b (which contains the treatment we're interested in). As expected the group
b proportion is higher than group
a. Say group
bs proportion is 0.007.
Can I use an exact binomial test on group
a with the alternative based on being greater than group
When I do this I get a p-value of 0.97, indicating that it's very unlikely that group
a will increase in conversion to trump group
bs current position.
I guess the question becomes one of power with this test? Any comments on the validity of this?