A/B test and SRS

Question

I was reading about Simple Random Sampling (SRS) from this lecture notes. It says that SRS, which actually is a sampling without replacement makes $x_i$ not independent.

Then I thought about A/B testing. When doing A/B testing for a product feature, usually users are sampled randomly without replacement, put in one of two buckets, and only one group will see the feature. Also, usually I analyse the result using t-test, which assumes independence of samples.

Is this a problem and what should be the workaround?

Answer 1

The author is talking about a finite sample correction. When A/B testing, the population is rather large, unless you have some website that is only accessible to only a limited number of people, so the correcting factor:

$$\frac{N-n}{N-1}$$

tends to $1$ and can be ignored, as long as the sample you draw for A/B testing ($n$) is supposedly smaller than the population of potential visitors to your site ($N$). Even if the sample was large compared to the population, how would you go about determining the population size of potential visitors to your website?

If anything, the usual unbiased estimator of variance has a larger denominator ($\frac{1}{n-1}$) than the finite-sample corrected version. In other words, the worst case scenario is an overestimation of the variance, resulting in slightly less power.

Answer 2

As a practitioner, I am thinking about this too. We need to figure out what is population/sample in each scenario and what is our estimand. Here is my conclusion:

• In Two Sample T-tests Scenario:

  We have 2 populations and draw 2 srs samples to compare population mean. Since our sample is srs, each sample is an unbiased represent of corresponding population, we use sample mean/variance to estimate population mean/variance and perform T-tests.

• In A/B Testing Scenario:

  We split users into 2 arms(control and treatment), and want to compare the mean difference of 2 arms. Thus, each arm itself is a population, instead of a srswor sample from total users. So we do not need to use finite sample correction to estimate means' population variance.