What's population, what's sample, in a website conversion hypothesis test?

Question

Context

I am a marketer; and I run marketing campaigns and website design tests a lot.

There are many tools - such as online A/B testing calculators - out there to help me test the statistical significance of the differences between two website designs or two marketing campaigns. I want to link that to what I’m learning now from an applied statistics class, with as much accuracy as possible (so far my impression has been since I’m taking an intro-level class, some concepts are papered over).

A hypothetical example

For example -

In a simple A/B test for a web page, say A is the control version, with the original design; B is the test version, with 1 element changed to test a hypothesis of a design. The success metric is the conversion rate from the website visitation into viewing product details (as a second step in the funnel).

Over a period of 1 month - 5000 people visited version A; 1850 out of these were converted. 5000 people visited version B; 2000 out of these were converted.

Since I’m interested in comparing proportions between two samples, I want to use a two-sample proportion z test to test the hypothesis that test design has a higher conversion than control design.

An image from online to illustrate the idea -

Source: Viget.com

My question

Now,

What are the two “populations” in this problem? And "samples"?

To me, the “date” factor is an interesting but tricky one. Within each date, let’s say Sept 07th, there are 2 “samples” of conversions. But zooming out to the entire testing period (1 month), there are 2 “samples” of conversions. I’m thinking, the population will never be complete as long as the page being tested is in use and welcoming new visitors. So here -

• Sample is defined using this “time” or “date” factor - sample refers to all the users in the test period; whereas
• Population on the other hand refers to all the users in the entire time period possible, as long as the page is live on the internet.

Is this correct? Am I selecting the right test to check statistical significance?

To really understand this, here is Another example -

It's slightly different. Let's say we are comparing covid incidence rates (number of daily cases/total population) of countries - say Germany and Spain - are the population vs. sample differentiated in a similar fashion, by the “time/date” factor? In this case, it seems to be even more so here as I couldn’t find another way to define the sample otherwise because we’re already looking at the entire population.

• Sample: incidence rates of Germany and Spain from Jan, 2020 to March 2022.
• Population: incidence rates of the two countries for all possible dates into the future.

Is this correct?

Answer 1

An A/B test is a (simple) randomized experiment; the concepts of population & sample in a randomized experiment are defined formally in framework called potential outcomes.

First think about just one user: they come to the website and either see version A or version B, but not both. One outcome is observed (factual); the other outcome is unobserved (counterfactual). Each user has the potential to see one or the other of the versions but not both.

So what are the two populations and the two samples in the A/B test?

One (conceptual) population consists of all users visiting the A webpage. We only have a sample from that population, the users that actually see version A.

The other population (also conceptual) consists of all users visiting the B webpage. Again, we only have a sample from that population because only a subset of users actually see version B.

The two populations are conceptual because -- while the test is running -- we cannot observe both the A population and the B population; for each user we make a choice, randomly, to observe either their behavior on the A version or the B version of the webpage.