Can we apply hypothesis testing to not-actively sampled groups?

Question

For argument's sake, in the below please assume that the hypothesis test we'd be considering would be a simple z-test to check whether an observed difference between two groups' means or proportions is statistically significant.

Looking at the literature for hypothesis testing, it almost always seems to assume that the groups we are applying a statistical test on have been actively sampled from a larger population. For example, you may be looking to find whether there is a difference between the average income of men vs women. For that, you would:

1. Sample a group of men and a group of women out of the larger well-defined population
2. Apply a two-tailed z-test to aim to reject the null hypothesis that the average income of the two groups is the same.

The literature on this is vast and well understood, and I feel confident applying statistical tests in such scenarios. But what if the groups have not been actively sampled from a larger well-defined population? Are statistical tests in-principle applicable then?

Please let me explain.

Imagine we work for a Retail business:

• In Week 1: 5,690 orders were placed with an AOV = £150 (AOV = Average Order Value)
• In Week 2: 6,152 orders were placed with an AOV = £155

Imagine the leadership team celebrating: "Hooray! We've increased AOV by £5 in a single week!". "Not so fast!", you may say, thinking to apply a z-test to verify that this difference didn't just happen by chance, but it's actually statistically significant. In the same way that you can flip an unbiased coin 6 times and get 6 consecutive HEADS, similarly the AOV may have increased just by chance because a small bunch of customers happened to spend lots of money on the same week.

Assuming all test's assumptions are satisfied: Can we in-principle apply a statistical test in this scenario?

Notice that the two groups were not actively sampled from a larger population. In Week 1, the group consists of all 5,690 orders that were placed that week. There is no larger "population" which contains additional orders that we could have sampled from. We also didn't actively select the orders either, as customers shopped on their own free will. Same with Week 2.

In this light, of not being able to define a larger population, when the two groups in question is "all we have".. is it possible to in-principle apply a statistical test to test whether the observed difference happened by chance?

Intuitively my answer would be 'yes', but searching through the literature I can only find applications of Hypothesis Testing in cases where a well-defined larger population exists.

So do you think statistical tests are applicable? If yes, can you point me to relevant literature that confirms that?

I'd be forever in your debt :)

Answer 1

I am not familiar with the term "not-actively sampled groups". But let's assume that the main issue for you is that you think there is no "larger population." and you can't use the law of large numbers.

1. As a general rule, I suggest you always prove that "all test's assumptions are satisfied". Devil is in the details
2. if you define the population as "people who bought something on the 4th July of 1981", then you have a discrete distribution, and you don't need to estimate the expected value because you already have a population.
3. If you have two expected values for two populations, there is no need for a "z-test" because these are not estimates.

More useful population:

4. Let's define our population as "people who bought something."
5. Each day for that definition is a sample, and we don't know the real expected value for the population
6. We perform a z-test to check if they are from the same population

The main problem here is that it would be hard to prove one of the assumptions for CLT (the foundation of the z-test): independent and identically distributed (iid). In real-world problems, you have seasonality, trends and occasional events that impact the sample. But it is another question.

I hope that it helped you to understand the idea behind population in split-testing!