I need an advice regarding the evaluation of the outputs for A/B testing. I have two groups of users, each one contains 1000 users. The group A was receiving emails from the system, while the group B was NOT receiving emails. Then I calculated the percentage of users who visited the web-site in both groups:
Group A Group B Visited web-site: 5% 2%
It seems that the users who were receiving emails from the system, were more likely to visit the web-site (5% of users in Group A), while those users who were NOT receiving emails, visited the web-site in less cases.
Now I want to statistically prove these results. So, I want to be sure that it's not a random result (i.e. to reject the null hypothesis9. Which statistical tests should I better use?
If it's necessary for the particular statistical test, I can also have more detailed results distributed over days, e.g.:
Group A Group B Visited web-site: Day 1 5.2% 2.1% Visited web-site: Day 2 4.8% 1.8% Visited web-site: Day 3 5.2% 1.9%