7
$\begingroup$

I am having trouble understanding why the Central Limit Theorem (CLT) is applicable in A/B testing. As a beginner in statistics, I am trying to grasp the intuition behind it.

The CLT states that as we draw random samples from a population, the distribution of their means tends towards a normal distribution. However, in A/B testing, we only draw two samples, and their distribution is not necessarily guaranteed to be normal. Nonetheless, the difference between these two samples is guaranteed to be a normal distribution. My question is, how is the difference between these two samples constructed, and why is it guaranteed to be a normal distribution?

$\endgroup$
2
  • $\begingroup$ The central limit theorem applies to the limit of the distribution of the mean (with a suitable location and scale adjustment) as the sample size increases, assuming iid samples and finite mean and variance. For finite sample sizes, the distribution is not necessarily normally distributed though eventually it will become close. $\endgroup$
    – Henry
    Commented Mar 19, 2023 at 12:50
  • 1
    $\begingroup$ The word sample is ambiguous, which may be part of what's steering you wrong. If I measure the height of 100 randomly chosen people, some would call that one sample (of size 100) and some would say I took 100 samples. It's the second meaning that's used here. $\endgroup$ Commented Mar 20, 2023 at 8:44

4 Answers 4

11
$\begingroup$

Suppose you have two populations: A and B.

You draw samples from A: $a_1,a_2,...,a_n$

You draw samples from B: $b_1,b_2,...,b_n$

The actual values of $a$'s and $b$'s are just the numbers $0$'s and $1$'s, which represent success/failures.

Now let $\alpha$ and $\beta$ denote the averages of these samples, i.e. $$ \alpha = \frac{a_1+a_2+...+a_n}{n} \text{ and }\beta = \frac{b_1 + b_2 + ... + b_n}{n} $$

These quantities $\alpha,\beta$ represent the proportion of successes that you see in each population. For example, suppose your samples for A included a total of 30 times where $a=1$ and 70 times where $a=0$. Then $\alpha = .3$, and so you estimate that the success rate for population A is roughly 30 percent.

You can apply the CLT to $\alpha$ and $\beta$ since they are means from a population. You are correct that that $a$'s and $b$'s are not normally distributed. But the moment you start taking their means they become normally distributed.

As a follow up question you ask "why is their difference normally distributed"? Their difference is given by $\alpha - \beta$. It is a well-known theorem in probability, that if $\alpha$ and $\beta$ are normally distributed, and they are independent, then $\alpha - \beta$ is also normally distributed.

Do you require help how to determine the $\mu$ and $\sigma$ parameters for their difference?

$\endgroup$
1
  • 1
    $\begingroup$ Thanks for your insights. I was mistaken the distribution of the sample mean with the distribution of the sample, I understand that the CLT applies to the sample mean, even though we only have 1 data point for it. I was also confused as to how the difference is calculated, it is simply the difference between 𝛼 and 𝛽, not the difference between the 2 sample distributions (which I'm unsure on if/how it can be calculated). $\endgroup$
    – Nab
    Commented Mar 22, 2023 at 12:40
7
$\begingroup$

we only draw two samples

You can consider a sample of size $n$ as $n$ samples of size $1$.

The outcome can be seen as a sum of $n$ independent Bernoulli distributed variables (if the people in the sample are independent), also know as a binomial distribution.

the distribution of their means tends towards a normal distribution

The central limit theorem tells that in the limit of an infinite sample size the distribution of the normalised sum will approach a normal distribution. In practice this is used to argue that a finite sample will also approximately follow a Normal distribution.

In the special case of a Binomial distribution we can also use the De Moivre–Laplace theorem to argue that the distribution is approximately normal distributed.

Related: Statistics question: Why is the standard error, which is calculated from 1 sample, a good approximation for the spread of many hypothetical means?

$\endgroup$
1
  • $\begingroup$ "You can consider a sample of size 𝑛 as 𝑛 samples of size 1." This is not clear to me. With a sample size of 1, the mean would be the observation itself, and the distribution of the sample means would approximate the distribution of the population as the number of sample increases. Online results seem to suggest that a minimum sample size of 30 is required for the CLT to hold. $\endgroup$
    – Nab
    Commented Mar 22, 2023 at 12:47
3
$\begingroup$

The CLT states that as we draw random samples from a population, the distribution of their means tends towards a normal distribution.

Not quite, though maybe you have a book that says something vague and not particularly accurate like this. Such vagueness of language has misled you about what it is even referring to.

It's not the number of samples, but the number of observations in a sample. Sample sizes are very large in typical A/B tests, but see the later discussion, which explains why that might not be sufficient for the sort of variables commonly used in many A/B tests. First lets look at what the CLT says, or at least let us get a bit closer to a formal statement of it.

In particular (for a 'classical' CLT in mean-form), if $\bar{X}_n$ (n=1,2,3...) is a sequence of sample means of $n$ independent and identically distributed observations from a population with finite mean and variance ($\mu$ and $\sigma^2$), and $Z_n = \frac{\bar{X}_n-\mu}{\sigma/\sqrt{n}}$ is the standardized mean, then in the limit as $n\to\infty$ the (cumulative) distribution function, $F_n(z)$ of $Z_n$ converges to the standard normal cdf, $\Phi(z)$.

(Conveniently, this theorem - relating as it does to distribution functions - is in a form that is potentially relevant to evaluating tail probabilities.)

This would suggest as sample sizes become very large, the distribution function $F_n$ of a statistic $Z_n$ should eventually become close to that of a standard normal distribution. The CLT itself doesn't tell you how large that might need to be; it only talks about what happens in the limit as $n$ goes to infinity. In some situations (even when the CLT holds), that might need to be very large indeed.

In particular, you might consider very skewed distributions (which are very common in calculations like click-through-rates or purchase rates or whatever, and also in effectively continuous quantities such as time or money spent on a site) and see that sample means can sometimes remain clearly non-normal even when sample sizes are getting into the thousands.

Nonetheless, the difference between these two samples is guaranteed to be a normal distribution.

I presume you intend "difference between sample means" there (otherwise, where does the CLT come in? 'difference in samples' is not a test statistic until we define how to do that), but either way, this is wrong. Indeed if the original distributions were non-normal you can guarantee that the distributions of the sample means (and of their difference, given the usual assumptions) is not actually normal. However, it might in practice get quite close $-$ close enough that the normal will yield perfectly reasonable answers, except perhaps in the extreme tail $-$ if only the sample size were large enough. Large enough, that is, given the particular situation you're in and your particular sense of what might be close enough.

My question is, how is the difference between these two samples constructed, and why is it guaranteed to be a normal distribution?

Look to the specific statistic you're using in the test. You don't mention which it is, and in A/B testing there's at least two distinct situations that are commonly involved and many more which might be possible. For both those common cases the statistic at least has a numerator with the form of a difference of means.

However, the CLT alone is not sufficient for the whole statistic in either case. A suitable argument for a t-test (e.g. if you were testing say, time or money spent at a site) or a z-test (as an approximation of a binomial test of proportions) would require additional argument, since the denominator is also a random variable. Such an argument is possible (e.g. by invoking Slutsky's theorem).

$\endgroup$
-3
$\begingroup$

Your question is full of incoherent statements.

The CLT states that as we draw random samples from a population, the distribution of their means tends towards a normal distribution.

In math, "tends to" is used to refer to a limit, and a limit always has some independent variable, and we take the limit as that variable goes to some value. Furthermore, a limit requires some norm and/or topology. Real numbers have the norm of the absolute value. PDFs do have norms, but there are more than one, so to be rigorous, one should specify one. So your statement of the CLT does not constitute a clear mathematical statement. And while one could infer some more rigorous statement, such that you mean "the $L^2$ norm of the distribution of their means minus a normal distribution goes towards 0 as the sample size goes to infinity", that still wouldn't be correct, because there is no one normal distribution that it goes towards. You have to take the z-score for it to go to a particular normal distribution.

However, in A/B testing, we only draw two samples, and their distribution is not necessarily guaranteed to be normal.

This also is not a precise statement. The normal distribution is a continuous distribution. A sample is a set of discrete values. What does it mean to compare them?

Nonetheless, the difference between these two samples is guaranteed to be a normal distribution.

Difference? A sample is a set of observations. How do you take the "difference" between two sets? There is the "set difference" of everything in one that isn't in the other, but how would that be normal? Perhaps you mean "the difference between their means". If so, you should be more precise. Furthermore, the mean is a particular number, not a distribution.

Precision is very important in mathematics. Yes, mathematicians speak loosely in some contexts, but if you're having trouble understanding something, that's not an appropriate context to be using casual language. You're asking people to explain something to you, and requiring them to make inference after inference as to what you mean.

The core issue in your question seems to be the statement "However, in A/B testing, we only draw two samples, and their distribution is not necessarily guaranteed to be normal." The distribution of sample means is approximately normal for large sample size, so your apparent intended statement is false.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.