For argument's sake, in the below please assume that the hypothesis test we'd be considering would be a simple z-test to check whether an observed difference between two groups' means or proportions is statistically significant.
Looking at the literature for hypothesis testing, it almost always seems to assume that the groups we are applying a statistical test on have been actively sampled from a larger population. For example, you may be looking to find whether there is a difference between the average income of men vs women. For that, you would:
- Sample a group of men and a group of women out of the larger well-defined population
- Apply a two-tailed z-test to aim to reject the null hypothesis that the average income of the two groups is the same.
The literature on this is vast and well understood, and I feel confident applying statistical tests in such scenarios. But what if the groups have not been actively sampled from a larger well-defined population? Are statistical tests in-principle applicable then?
Please let me explain.
Imagine we work for a Retail business:
- In Week 1: 5,690 orders were placed with an AOV = £150 (AOV = Average Order Value)
- In Week 2: 6,152 orders were placed with an AOV = £155
Imagine the leadership team celebrating: "Hooray! We've increased AOV by £5 in a single week!". "Not so fast!", you may say, thinking to apply a z-test to verify that this difference didn't just happen by chance, but it's actually statistically significant. In the same way that you can flip an unbiased coin 6 times and get 6 consecutive HEADS, similarly the AOV may have increased just by chance because a small bunch of customers happened to spend lots of money on the same week.
Assuming all test's assumptions are satisfied: Can we in-principle apply a statistical test in this scenario?
Notice that the two groups were not actively sampled from a larger population. In Week 1, the group consists of all 5,690 orders that were placed that week. There is no larger "population" which contains additional orders that we could have sampled from. We also didn't actively select the orders either, as customers shopped on their own free will. Same with Week 2.
In this light, of not being able to define a larger population, when the two groups in question is "all we have".. is it possible to in-principle apply a statistical test to test whether the observed difference happened by chance?
Intuitively my answer would be 'yes', but searching through the literature I can only find applications of Hypothesis Testing in cases where a well-defined larger population exists.
So do you think statistical tests are applicable? If yes, can you point me to relevant literature that confirms that?
I'd be forever in your debt :)