Advice on validating statistical significance of results of AB testing

Question

I need an advice regarding the evaluation of the outputs for A/B testing. I have two groups of users, each one contains 1000 users. The group A was receiving emails from the system, while the group B was NOT receiving emails. Then I calculated the percentage of users who visited the web-site in both groups:

                   Group A  Group B
Visited web-site:  5%       2%

It seems that the users who were receiving emails from the system, were more likely to visit the web-site (5% of users in Group A), while those users who were NOT receiving emails, visited the web-site in less cases.

Now I want to statistically prove these results. So, I want to be sure that it's not a random result (i.e. to reject the null hypothesis9. Which statistical tests should I better use?

If it's necessary for the particular statistical test, I can also have more detailed results distributed over days, e.g.:

                          Group A  Group B
Visited web-site: Day 1   5.2%     2.1%
Visited web-site: Day 2   4.8%     1.8%
Visited web-site: Day 3   5.2%     1.9%

Answer 1

As pointed out by @IWS, you can use chi-square test for that purpose. Here, a call to the R chi-square test function with your settings gives the following:

prop.test(c(0.05, 0.02) * 1e3, c(1e3, 1e3), alternative = 'two.sided', conf.level = 0.95)

    2-sample test for equality of proportions without continuity correction
data:  c(0.05, 0.02) * 1000 out of c(1000, 1000) 
X-squared = 13.323, df = 1, p-value = 0.000418
alternative hypothesis: two.sided
95 percent confidence interval: 0.01294504 0.04705496
sample estimates: prop 1 prop 2 
   0.05   0.02

Concerning the test, the main result is the p-value of 0.000418, which means that your null hypothesis is rejected with high confidence, i.e. the user behavior are not the same in both versions.

Instead of just running a statistical test, it also computes the confidence interval around the difference between the two rates of visit. Here, the 95% confidence interval is the following: [0.01294504, 0.04705496]. As it does not contain 0, it leads to the same conclusion as previously. However, it gives you another angle on it, because you see that the interval is quite far from 0.

To measure, the confidence you should have in these previous results, a good thing is to estimate the power of your statistical test given your confidence level (set to 95% in the above), i.e. 1 minus type II error, i.e. the proportion of chances to conclude that both behaviors are the same while they are not. For that, you can do:

power.prop.test(p1=0.05, p2=0.02, n=1e3, alternative='two.sided', sig.level=0.05)
     Two-sample comparison of proportions power calculation 
              n = 1000
             p1 = 0.05
             p2 = 0.02
      sig.level = 0.05
          power = 0.9550406
    alternative = two.sided
NOTE: n is number in *each* group

Here, the test power is approximately 0.955, which is high and generally considered enough to conclude.