# Hypothesis testing: understanding test statistics and their sampling distribution

I am struggling a bit with the real meaning of the test statistic and their sampling distribution, but I cannot find my answer. So, I am looking for clarifications. Sorry in advance if it seems very basic.

To illustrate my question and to keep it simple, I will take the example of a one-sample t-test: $$t= \frac{\overline{x} - \mu_0}{se}$$

When we say that test statistics have their sampling distributions, does it mean that (in this case), the sampling distribution of the t-test is drawn from sampling distribution of the mean $$\overline{X}$$? Or is it drawn from the distribution of the null hypothesis assuming the mean of our sample $$\overline{x}$$ is fixed?

• If the $X_i$ are iid $N(\mu_0,\sigma^2)$ then $\bar X - \mu_o$ is a random variable (with a normal distribution). So too is the sample standard error (with a scaled $\chi$ distribution), and they are independent. This means their ratio is also a random variable (with a $t$ distribution as the sampling distribution). Aug 25, 2021 at 10:51
• Thank you @Henry ... this is very helpful. I am conscious it might seem trivial but it does help in understanding the full picture! Aug 25, 2021 at 11:02

I feel it could bring some value to illustrate @Lewian's answer and @Henry's comment for people like me who are more "visual". I will take the example of the same t-test used in the question $$\frac{\bar X-\mu_0}{S/\sqrt{n}}$$, but focusing more on big picture.

Let's assume that we have a random sample (people's height in cm for example) of size $$n = 20$$ from a given country population. We want to determine whether the population mean is different from $$\mu_0 = 168$$ (for the sake of the exercise). We will simulate our sample in R as follows:

set.seed(1)
n <- 20
X <- sample(150:200, n, replace=TRUE)
X


X = {153, 188, 150, 183, 172, 192, 163, 167, 200, 182, 170, 170, 191, 195, 159, 156, 158, 164, 170, 186}

Numerator $$\overline{X} - \mu_0$$

Our sample is only one possible sample out of the many we could have drawn from our population. It has a mean $$\overline{x}$$ (actual value) and a standard deviation $$s$$ (notation for both in lower case that represent observed data). If we imagine that we could get a significant number of random samples (of the same size) from our population of interest, we would be able to calculate the mean for each of them. This distribution of sample means is called the sampling distribution of the means. In our t-test, $$\overline{X}$$ denotes the random variable that represents this distribution.

The Central Limit Theorem states that, given a sufficiently large sample size, the sampling distribution of the mean for a variable will approximate a normal distribution. We can empirically illustrate that by simulating this sampling distribution of the mean through bootstrapping : Since $$\mu_0$$ is fixed (in our t-test we assume that the null hypothesis is true), then $$\overline{X}-\mu_0$$ is also a random variable. Same distribution as above (normal), but centered on the effect (the difference between the population value and the null hypothesis - illustration example below). Denominator $$S/\sqrt{n}$$

The denominator is actually the standard error of the mean which measures the variability of sample means in the sampling distribution of means. $$S$$ is the estimate of the standard deviation of the population. It is also a random variable and $$S^2$$ follows a chi-square distribution.

Ratio: t-test $$\frac{\overline{X} - \mu_0}{S/\sqrt{n}}$$

The numerator is a random variable with a normal distribution, the denominator is a random variable with a scaled chi-square distribution. Both are independent, so it means that their ratio is also a random variable that follows a t-distribution (see "Characterization" in https://en.wikipedia.org/wiki/Student%27s_t-distribution).

Not sure if it helps... but I tried ;-)

• Thanks @Pitouille! It helps indeed! Sep 3, 2021 at 10:39

The meanings that you propose in your question are not wrong but problematic, because you make imprecise and somewhat confused use of terminology. You may get more insight carefully distinguishing the different meanings of the word "mean", see below.

The null hypothesis model is $$X_1,\ldots,X_n$$ i.i.d. with $$X_i\sim N(\mu_0,\sigma^2)$$. A very important distinction in statistics is between a true, but unobserved parameter value of the underlying distribution we are interested in, and an observable test statistic that we use to estimate it or do inference about it. In English, it seems that mostly, somewhat confusingly, the term "mean" is used for both the true underlying unobserved expected value $$\mu_0$$ and for the observed data mean $$\bar x$$, which is a fixed value and modelled as realisation of the random variable $$\bar X$$. Note that there are three kinds of means now, $$\mu_0, \bar x, \bar X$$, the latter two distinguished by whether we have already made the observations and computed the mean (that's $$\bar x$$, a fixed number but only fixed after having observed the data), or we're talking about the general method to compute the mean from our observations whatever they are (that's $$\bar X$$, a random variable).

The sample distribution of the t-statistic is derived from the null hypothesis distribution as you state, however the "fixed mean" of it is the underlying true $$\mu_0$$, not the $$\bar x$$.

Assuming this, there is a sampling distribution of the random variable $$\bar X$$ as computed from the $$X_1,\ldots,X_n$$, referred to as "sampling distribution of the mean", but note this is $$\bar X$$, not $$\bar x$$, because the latter one is a specific mean value of a specific sample.

The sampling distribution of the t-statistic $$\frac{\bar X-\mu_0}{S/\sqrt{n}}$$ (note my use of capital letters here; we can only speak of the sampling distribution of random variables, which, if you want, stand for methods of computing things rather than already computed fixed values) is strongly mathematically related to the sampling distribution of $$\bar X$$, but some more math is required to bring in the connection to the estimated standard deviation $$S$$, which is another random variable (if required, one can also use lower case $$s$$ for the actual value it takes after having collected the sample, and $$\sigma$$ for the unobserved underlying true value of the sd). The wording "drawn from the sampling distribution of the mean" isn't really appropriate here; the idea is that random variables $$X_1,\ldots,X_n$$ are drawn from the original null distribution, and $$\bar X$$ and $$S$$ are computed from these random variables, which implies that their sampling distribution has the stated form, such as $$t_{n-1}$$ for the test statistic, and $$N(\mu_0,\frac{\sigma^2}{n})$$ for the mean random variable $$\bar X$$.

I know I am somewhat redundant here, but I hope it helps to review the key difference between true parameters, random variables, and values computed from samples at each step.

• Thank you @Lewian, this is very helpful... I am conscious my question was very basic but after reading some of the posts here, I guess it might be useful for other people as well! Sep 1, 2021 at 11:05
• Just to make sure that my understanding is correct, in the formula $\frac{\bar X-\mu_0}{S/\sqrt{n}}$, capital S is the standard deviation of the sampling distribution of the mean... right @Lewian? Then the actual test is made on the values we observed in our sample. Sep 1, 2021 at 11:25
• Capital $S$ is the estimator of the standard deviation $\sigma$ of the original sample (called $\hat\sigma$ on the Wikipedia page on the t-test; otherwise my notation is in line with that). This is not the standard deviation of the sampling distribution of the mean (which is $\frac{\sigma}{\sqrt{n}}$, and is not a random variable, whereas $S$ is). Sep 1, 2021 at 13:04
• Not sure where you got your formula for the t-statistic from. Apart from having used lower case letters (which is OK if you talk about the resulting value rather than a random variable), you use "se", which seemingly refers to the standard error, which is an estimator of the standard deviation of the sampling distribution of the mean. This is not wrong but usually written out as $\frac{s}{\sqrt{n}}$ or $\frac{S}{\sqrt{n}}$ as random variable, corresponding to the fact that the standard deviation of the mean is $\frac{\sigma}{\sqrt{n}}$, see above. Sep 1, 2021 at 13:11
• As I had written before, Wikipedia writes $\hat\sigma$ for what I use $S$ (random variable) or $s$ (observed value), so it's the same thing, and it estimates $\sigma$ (the parameter), not $\hat\sigma$ (which itself is an estimator). It seems I worded my earlier comment somewhat ambiguously as I can see how you could think that I meant the parameter by $\hat\sigma$, but no, it's my $S$, sorry. Sep 1, 2021 at 14:09