Generally when reading about statistics in A/B testing, most hypothesis tests are done assuming that the distribution of the sample mean follows a normal distribution where the standard deviation is given by the $\frac{\sigma}{\sqrt{n}}$.

In general statistics when taking large samples relative to the population, for example with a sample size of 500 and a population size of 1000, we need to calculate the variance with a fixed population correction factor which I believe is $(1-\frac{n}{N})\frac{\sigma^2}{n}$. The standard deviation is then the square root.

My question is why we do not often use the fixed population correction factor in online A/B tests? Is it because the population sizes are mostly unknown, as the population is generally the number of future users? Is it because the standard deviation of the sample mean with the fixed population correction factor is always larger than that of the standard deviation with the correction factor (I assume this is True from the formula) and therefore it will always be a conservative bound? Is it because we are often working with continuous metrics, where the probability of sampling the same value again has probability 0?


1 Answer 1


This is a great question. The short answer is, I think, what you suggested, i.e. typically we want to use an a/b test to infer something about a larger (super) population, say of potential users, that, theoretically is infinite.

The long answer is more complicated but more interesting.

TL;DR. For finite (rather than infinite) causal inference, observations in treatment and control group are not independent so while you could use the finite population correction you'd also have to include a covariance term but the covariance term generally can't be estimated. 1) Dropping the finite population correction is a conservative way to deal with this problem. 2) Under the assumption of the sharp null this can be justified. 3) The correlation coefficient can be used to justify a (generally) smaller estimate for variance.

General case

Define the population size as $n$ and any subset of the population as $n_t$, $n_c$, etc. Define $W_1, W_2, ... W_n$ where $W_i$ is a random variable that indicates inclusion ($W_i = 1$) or exclusion ($W_i = 0$) from the sample. Furthermore, define $y_1, y_2, ... y_n$ as our variable of interest. We denote $y$ with the lower-case because per the design-based approach to sampling (and to causal inference) $y$ is fixed.

For a simple random sample without replacement, the sample mean, $\frac{1}{n_t} \sum^n_{i = 1} W_i y_i$, is an unbiased estimator for the population mean. It can be shown that

$$ Var(\frac{1}{n_t} \sum^n_{i = 1} W_i y_i) = \frac{S_t^2}{n_t} \cdot (1 - \frac{n_t}{n}) $$

The second term, $(1 - \frac{n_t}{n})$, is referred to as the finite population correction. If you work through the derivation, you will see that it arises because, without loss of generality, in sampling without replacement the first unit is not independent of the second unit. (The first unit is selected with probability $\frac{n_t}{n}$ and the second unit is selected with probability $\frac{n_t - 1}{n -1}$ if $W_1 = 1$ and with probability $\frac{n_t}{n -1}$ if $W_1 = 0$.) Of course, as $n \to \infty$, $\frac{n_t}{n} \to 0$, and the correction disappears because the dependence disappears.

Now we know that

$$ Var(X - Y) = Var(X) + Var(Y) - 2 \cdot Cov(X, Y) $$ and that when $X$ and $Y$ are independent the covariance term is zero. This is the case for causal inference for a infinite (super) population.

Causal inference for finite populations

Causal inference for a finite population is a special case.

Define $t_1, t_2, ... , t_n$ and $c_1, c_2, ... , c_3$ as the value for the variable of interest for a unit if it is treated or not treated respectively (or assigned to treatment a or b). We denote both vectors with the lower-case because, from the perspective of the potential outcome framework, these values are fixed.

We'd like to recover the average treatment effect (difference in means)

$$ \frac{1}{n} \sum^n_{i=1} t_i - \frac{1}{n} \sum^n_{i=1} c_i $$

but this is impossible because we have to assign some units to treatment (and for those units we don't observe the value of $c$) and other units to control (and for those units we don't observe the value of $t$).

This random assignment is identical to the mechanism of random sampling defined above. Thus, we estimate the average treatment effect as

$$ \bar{t} - \bar{c} = \frac{1}{n_t} \sum^{n}_{i=1} W_i t_i - \frac{1}{n_c} \sum^{n}_{i=1} (1 - W_i) c_i $$

but different assignments will (generally) produce different estimates.

$$ \begin{aligned} Var(\bar{t} - \bar{c}) &= Var( \frac{1}{n_t} \sum^{n}_{i=1} W_i t_i ) + Var(\frac{1}{n_c} \sum^{n}_{i=1} (1 - W_i) c_i) - 2 \cdot Cov(\frac{1}{n_t} \sum^{n}_{i=1} W_i t_i, \frac{1}{n_c} \sum^{n}_{i=1} (1 - W_i) c_i) \\ \end{aligned} $$

We have the first and second term from above. It can be shown that the covariance is $-\frac{1 }{n (n - 1)}\sum_{i=1}^n (t_i - \bar{t}) \cdot (c_i - \bar{c})$. Making these substitutions gives us

$$ \begin{aligned} Var(\bar{t} - \bar{c}) &= \frac{S_t^2}{n_t} \cdot (1 - \frac{n_t}{n}) + \frac{S_c^2}{n_c} \cdot (1 - \frac{n_c}{n}) + \frac{2 }{n (n - 1)}\sum_{i=1}^n (t_i - \bar{t}) \cdot (c_i - \bar{c}) \end{aligned} $$

The problem is that while we can estimate the first and second term, we generally have no way of estimating the last term because we never observe $t_i$ and $c_i$ together rather only $t_i$ but not $c_i$ or vice versa.

The most common approach is to drop the finite population correction and the covariance. The resulting estimator for the variance is then $\frac{S_t^2}{n_t} + \frac{S_c^2}{n_c}$ which is generally conservative.

Variance for sharp null

Alternatively, this approach can be justified under the sharp null hypothesis of no treatment effect on any unit by using the variance of the unit-level treatment effects which we denote by $S_{ct}^2$.

$$ \begin{aligned} S_{ct}^2 &= \frac{1}{n-1} \sum_{i=1}^n [ (t_i - \bar{t}) - (c_i - \bar{c})]^2 \\ &= \frac{1}{n-1} \sum_{i=1}^n [ t_i - \bar{t} ]^2 + \frac{1}{n-1} \sum_{i=1}^n [ c_i - \bar{c}]^2 - \frac{2}{n-1} \sum_{i=1}^n (t_i - \bar{t}) (c_i - \bar{c}) \\ &= S_t^2 + S_c^2 - \frac{2}{n-1} \sum_{i=1}^n (t_i - \bar{t}) (c_i - \bar{c}) \end{aligned} $$ which means that $\frac{2}{n-1} \sum_{i=1}^n (t_i - \bar{t}) (c_i - \bar{c}) = S_t^2 + S_c^2 - S_{ct}^2$. Using this substitution for the equation above and the fact that $n_t = n - n_c$ and $n_c = n - n_t$ we simplify further but first note that

$$ \frac{S_t^2}{n_t} \cdot (1 - \frac{n_t}{n}) = \frac{S_t^2n - S_t^2n_t}{n n_t} = \frac{S_t^2n - S_t^2(n - n_c)}{n n_t} = \frac{S_t^2n_c}{n n_t} $$

and likewise for $\frac{S_c^2}{n_c} \cdot (1 - \frac{n_c}{n})$. This gives us

$$ \begin{aligned} Var(\bar{t} - \bar{c}) &= \frac{n_c S^2_t}{n n_t} + \frac{n_t S^2_c}{n n_c} + \frac{1}{n}(S_t^2 + S_c^2 - s_{ct}^2) \\ &= \frac{n_c S^2_t}{n (n - n_c)} + \frac{S^2_t}{n} \frac{(n - n_c)}{(n - n_c)} + \frac{n_t S^2_c}{n (n - n_t)} + \frac{S_c^2}{n} \frac{(n - n_t)}{(n - n_t)} - \frac{s_{ct}^2}{n} \\ &= \frac{n_c S^2_t + S_t^2 n - S_t^2 n_c}{n (n - n_c)} + \frac{n_t S^2_c + S_c^2 n - S_c^2 n_t}{n (n - n_t)} + - \frac{s_{ct}^2}{n} \\ &= \frac{S^2_t}{n_t} + \frac{S^2_c}{n_c} - \frac{S_{tc}^2}{n} \end{aligned} $$ Returning to the definition of $S^2_{ct}$ note that if a treatment has no effect for each unit, then $(t_i - \bar{t}) - (c_i - \bar{c}) = 0$ so the last term will equal zero.

Bounds on variance

Another approach is to utilize the correlation coefficent. Because the correlation coefficient is bound between $[-1, 1]$ we can establish an upper bound on the variance by substituting $\rho_{ct} = 1$.

$$ \begin{aligned} Var(\bar{t} - \bar{c} | rho = 1) &= \frac{n_c S^2_t}{n n_t} + \frac{n_t S^2_c}{n n_c} + \frac{2}{n(n-1)} \sum_{i=1}^n (t_i - \bar{t}) (c_i - \bar{c}) \\ &= \frac{n_c S^2_t}{n n_t} + \frac{n_t S^2_c}{n n_c} + \frac{2}{n (n-1) s_t s_c} \sum_{i=1}^n (t_i - \bar{t}) (c_i - \bar{c}) \cdot s_t \cdot s_c \\ &= \frac{n_c S^2_t}{n n_t} + \frac{n_t S^2_c}{n n_c} + \frac{2}{n} \rho_{ct} \cdot s_t \cdot s_c \\ &= \frac{n_c S^2_t}{n n_t} + \frac{n_t S^2_c}{n n_c} + \frac{2 (1)}{n} \cdot s_t \cdot s_c \\ \end{aligned} $$ This approach typically produces a smaller estimate for variance than $\frac{S^2_t}{n_t} + \frac{S^2_c}{n_c}$. However, the difference is usually marginal.

The result for finite population causal inference is due to Neymar. For an English translation see of the key paper here. For more on bounds for the variance see "Sharp bounds on the variance in randomized experiments," by Aronow, Green, and Lee. For a thorough introduction to this discussion see Causal Inference by Imbens and Rubin.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.