I am a marketer; and I run marketing campaigns and website design tests a lot.
There are many tools - such as online A/B testing calculators - out there to help me test the statistical significance of the differences between two website designs or two marketing campaigns. I want to link that to what I’m learning now from an applied statistics class, with as much accuracy as possible (so far my impression has been since I’m taking an intro-level class, some concepts are papered over).
A hypothetical example
For example -
In a simple A/B test for a web page, say A is the control version, with the original design; B is the test version, with 1 element changed to test a hypothesis of a design. The success metric is the conversion rate from the website visitation into viewing product details (as a second step in the funnel).
Over a period of 1 month - 5000 people visited version A; 1850 out of these were converted. 5000 people visited version B; 2000 out of these were converted.
Since I’m interested in comparing proportions between two samples, I want to use a two-sample proportion z test to test the hypothesis that test design has a higher conversion than control design.
An image from online to illustrate the idea -
What are the two “populations” in this problem? And "samples"?
To me, the “date” factor is an interesting but tricky one. Within each date, let’s say Sept 07th, there are 2 “samples” of conversions. But zooming out to the entire testing period (1 month), there are 2 “samples” of conversions. I’m thinking, the population will never be complete as long as the page being tested is in use and welcoming new visitors. So here -
- Sample is defined using this “time” or “date” factor - sample refers to all the users in the test period; whereas
- Population on the other hand refers to all the users in the entire time period possible, as long as the page is live on the internet.
Is this correct? Am I selecting the right test to check statistical significance?
To really understand this, here is Another example -
It's slightly different. Let's say we are comparing covid incidence rates (number of daily cases/total population) of countries - say Germany and Spain - are the population vs. sample differentiated in a similar fashion, by the “time/date” factor? In this case, it seems to be even more so here as I couldn’t find another way to define the sample otherwise because we’re already looking at the entire population.
- Sample: incidence rates of Germany and Spain from Jan, 2020 to March 2022.
- Population: incidence rates of the two countries for all possible dates into the future.
Is this correct?