Why is the t-test designed for small samples?

Question

I've read somewhere that the t-test is designed for small samples, why is that the case? What if I use a big sample out of a population to conduct the t-test? Would that be an issue or have potential implications?

Answer 1

There is fundamental misconception in your question and in several of the answers you have received so far.

The choice of $t$-test or $z$-test is about the nature of the test statistic, specifically whether the variance of the sampling distribution of the sample mean is known ($z$-test) or unknown and must be estimated from the sample ($t$-test).

How well either of these tests perform for statistical inference depends on an underlying assumption that the data are realizations from a normal distribution. However, because of the central limit theorem, the larger the sample size, the more robust both of these tests will be with respect to deviations from normality, since with increasing sample size, the CLT states that the sample mean (under certain regularity conditions) will asymptotically tend toward a normal distribution.

These two things (known versus unknown variance, and the relationship between sample size and asymptotic normality) are commonly conflated in statistical practice. They are not the same thing. To illustrate:

• If the data are truly observations from a normal distribution, but the mean of the distribution is unknown and the variance is known, the sample size is irrelevant--even a sample size of $n = 1$ is enough to say that the test statistic is exactly normally distributed. Inference on a very small sample size is still possible using a $z$-statistic.
• If the data are truly observations from a normal distribution as above, but the variance is unknown, then the test statistic $$T \mid H_0 = \frac{\bar X - \mu_0}{S/\sqrt{n}}$$ is exactly $t$-distributed, because $\bar X$ is exactly normally distributed and $S^2$ is exactly chi-squared distributed.
• If the data are not generated from a normal distribution, but the variance is known, then whether a $z$-test may be used depends on the sample size. Specifically, if the sample size is large enough (where "large" depends on the extent of deviation from normality but is often unknown in practice), then the $z$-test may be an acceptable approximation. Otherwise, a nonparametric (i.e., distribution-free in its assumptions) test of location should be used in order to avoid potential failure to control Type I error. But the $t$-test should not be used here.
• If the data are not generated from a normal distribution, and the variance is unknown, then the choice is between a $t$-test or a nonparametric test, depending on the extent of deviation from normality and the sample size. If the deviation from normality is minor, then a $t$-test is reasonably robust and may be justified even when the sample size is small. But in the small sample case with larger deviations from normality, a nonparametric test should be used. If the sample size is large, to the point where the CLT kicks in, there is generally little difference between a $t$-test and $z$-test because the degrees of freedom will generally be high enough that the critical values will be nearly identical.

I hope this clarifies the situation, because a LOT of people who use univariate hypothesis tests fail to get this right.

Answer 2

I think this is a slight mis-reading of the place where the quote came from. It is perfectly true that when he was inventing the test which bears his pseudonym Gosset was trying to solve the problem of small samples because that was what he encountered in his work. So he designed them for small samples. That historical fact does not mean they do not work for large samples.

In his original paper he outlined the problem

In routine work there are two ways of dealing with this difficulty: (1) an experiment may he repeated many times, until such a long series is obtained that the standard deviation is determined once and for all with sufficient accuracy. This value can then be used for subsequent shorter series of similar experiments.

and points out

There are other experiments, however, which cannot easily be repeated very often; in such cases it is sometimes necessary to judge of the certainty of the results from a very small sample, which itself affords the only indication of the variability. Some chemical, many biological, and most agricultural and large- scale experiments belong to this class, which has hitherto been almost outside the range of statistical inquiry.

Gosset himself was working with the brewers, Guinness, and faced the problem that he mentions in that paragraph of being unable to run large-scale repetitions and even if he could have repeated the plots many times there would almost certainly have been drift over time.

The quotes are from his 1908 article in Biometrika "The probable error of a mean" which has been widely reprinted.

Answer 3

The question $t$-tests vs $z$-tests? suggested by @user2974951 seems relevant but perhaps the answer there remains a bit abstract. I like to see it in this way, possibly not 100% correct, though.

With small sample sizes, it's relatively likely that one of the two samples will receive a slightly extreme datapoint that will skew its mean and make you think that there is a real difference between the two samples when instead nothing is going on. To correct for such small-size effect you allow more extreme values of the difference between samples to be relatively likely before claiming "significance of difference". I.e. the t-distribution, against which you map the value of the t-statistics to obtain the p-value, has fatter tails than the normal distribution. As the sample sizes increases, this correction becomes negligible and you don't need to worry about using Normal or T-distribution (or T-test vs Z-test).

Here's an example using simulation. We compare a treatment to a control where in fact there is no difference between to two. We simulate the data from a normal distribution, so no complications about data normality, unequal variance, etc. Since the null-hypothesis holds, by repeating the sampling and testing many times, we expect ~5% of the tests to have p < 0.05. If we use t.test this is actually the case but if we use Z-test (i.e. we don't account for small sample size) we get many more false positives. Here, the sample size is just N=3:

    set.seed(1234)
    N <- 10000
    t.pvalue <- rep(NA, N)
    z.pvalue <- rep(NA, N)
    
    z.test <- function(x, y) {
        diff <- mean(x) - mean(y)
        sd_diff <- sqrt(var(x)/length(x) + var(y)/length(y))
        z <- diff/sd_diff
        p <- (1 - pnorm(abs(z))) * 2
        return(p)
    }
    
    my.t.test <- function(x, y) {
        # Custom t-test instead of R's to make things 
        # more transparent
        stopifnot(length(x) == length(y))
        diff <- mean(x) - mean(y)
        sd_diff <- sqrt(var(x)/length(x) + var(y)/length(y))
        z <- diff/sd_diff
        df <- (2 * length(x)) - 2
        p <- (1 - pt(abs(z), df= df)) * 2
        return(p)
    }
    
    for(i in 1:N) {
        treat <- rnorm(n= 3, mean= 10)
        ctrl <- rnorm(n= 3, mean= 10)
        t.pvalue[i] <- my.t.test(treat, ctrl)
        z.pvalue[i] <- z.test(treat, ctrl)
    }
    sum(t.pvalue < 0.05) # 499 "false positives" as expected
    sum(z.pvalue < 0.05) # 1234 "false positives"

The T-test behaves as expected giving 499 cases with p < 0.05, the Z-test gives many more, 1234. If you re-run the simulation with a larger N, say, rnorm(n= 30, ...) the distortion disappears.

You can also check the distribution of pvalues noting that under the null hypothesis the distribution should be uniform (flat) between 0 and 1. This is the case for the T-test but not for the Z-test:

    par(mfrow= c(1, 2))
    hist(t.pvalue)
    hist(z.pvalue)

Answer 4

Short answer: When people say the t test (that is, a test based on a t distribution) is "designed" for small samples what they mean is that if you have a small sample then a t test is going to be be more accurate than a "z test" (which is based on a normal distribution). If a t test and a z test ever give you different answers (which will only happen with small sample sizes), you should prefer the t test answer. But as the sample size increases the two tests approach each other so it doesn't matter which you use.

Longer answer: The underlying logic of both of these tests is that we know that the ERRORS associated with using a random sample (rather than the whole population) are always going to to have a particular bell shaped distribution. This is true regardless of what the distribution of the variables in question looks like. If we know the shape of that distribution we can use it to figure out how likely it is that two estimates that look different really are different in the underlying population (which is what the t test is trying to figure out).

Now, when sample sizes are "large" these sampling errors are described by the famous normal curve. But as the sample size gets smaller the normal curve ends up being too "skinny" to describe how the errors actually work in reality. So we use this other distribution - the t distribution - which is like the normal curve but with some added "fatness" that depends on the sample size. This means that there isn't just one t distribution - there are an infinite number, one for every possible sample size. But the bigger the sample size is the closer the t and normal distributions look to each other, so it doesn't matter what you use, because they will give you the same answer. But if there is ever a situation in which the t and normal distributions DO give you a different answer then you should use the t distribution, because the fact that they are different means that your sample size is "small enough" for the normal curve to be inaccurate.

Answer 5

The simple answer is that in small samples you can't use the sample estimate of the variance as the variance itself, while in large samples you can due to it converging to the true value under realistic assumptions.

So, when you take a small sample of size $n$ of variable $x$, even if you suspect it is from normal distribution $x\sim\mathcal N(0,\sigma^2)$, and estimate its sample variance $\hat\sigma^2=s_n^2$ then try to standardize the variable as $z=\frac x {\hat\sigma}$, it doesn't make $z$ a standard normal variable. The reason is that $\hat\sigma\equiv\sqrt{s^2_n}\ne\sigma$. However, if your sample is "large", then $\hat\sigma\equiv\sqrt{s^2_\infty}=\sigma$, so $z\sim\mathcal N(0,1)$.

In small samples it happens so that $z\sim t_{n-1}$, but you know that $t_\infty=\mathcal N(0,1)$. So it works out.

Answer 6

Why is the t-test designed for small samples?

Because it is the small samples where the t-test makes a difference.

For large samples the t-distribution becomes similar to the normal distribution.

What if I use a big sample out of a population to conduct the t-test? Would that be an issue or have potential implications?

No, the t-test works for both large and small samples.

---

Background

The t-test is a hypothesis test that uses the difference of the observed sample mean $\bar{X}$ with the hypothesized mean $\mu$, and divides/standardizes this by the observed standard deviation. You get the test statistic $t = \frac{\bar{X}-\mu}{\hat \sigma / \sqrt{n}}$ which is not normally distributed but instead t-distributed* with degrees of freedom $\nu = n-1$ (where $n$ is the sample size). The trick of the t-test is to use this t-distribution instead of the normal distribution.

^{*Assuming normal distribution for the population, although it may still work very well for other population distributions.}

This is important for small samples. The normal distribution is not accurate when the sample size is small. For large samples, it does not matter (much). For large sample size, the t-distribution and the z-distribution become very much the same as you can see in the image below.

---

Additional note. The t-test is designed for cases where the sample is small and when the standard deviation is unknown. If the standard deviation is known, then the sample size does not matter. The issue with the t-test is the uncertainty in the estimate of the standard deviation (which becomes less for a larger sample size).

---

Related

• What is the rationale behind using the t-distribution?
• T-distribution versus normal distribution (sample means and linear inference)

Answer 7

Many statistical tests use some form of approximation, which makes them only applicable for sufficiently large sample sizes. But the t-test doesn't have this disadvantage. It is also exact for very small sample sizes, if its condition (normal distributed data) is sufficiently fulfilled. Of course you can use the t-test also for large sample sizes.

Answer 8

The difference between the t-test and z-test is that the t-test can be used with both small and large sample size, instead the z-test is used only with large sample size. But now you can ask me: Why is this so? Why we use z-test instead of using only t-test?

Well, the reason is simple: z-test is an approximation of the t-test and is valid under the assumption of a large sample size.

The z-test, for a fixed Confidence Level, has a fixed number of SDs from the mean, while in the t-test, for a fixed Confidence Level, the number of SDs from the mean depend also on the sample size; but the larger the size of the sample, the smaller the difference between the number of SDs used in the t-test and in the z-test; in other words, the larger the sample size, the thinner the t-test tails become until the 2 distributions match almost perfectly.

i.e. if we use both t-test and z-test on the same dataset with sample_size > 50 we will see no (relevant) difference between the z-test's p-value and the t-test one. if we use both t-test and z-test on the same dataset with sample_size = 3 we will se a huge difference between the two p-value.

For the last 2 question the answer is: No, neither issues nor practical implications because the difference between the 2, for large sample size, is irrelevant.

Answer 9

The t-test allows for the fact that the mean and standard deviation have been calculated from the data rather than known beforehand. It basically is a test on the normal distribution with a "fudge factor." The fudge factor is necessary because in a small sample the experimenter is unlikely to adequately sample the tails of the distribution. So the experimental standard deviation tends to be smaller than the true (or population's) standard deviation.

Tables for the t-test will indeed have a rows for larger sample sizes and a row for an infinite sample size. The infinite sample size is just the normal distribution itself. All -in-all to get a reasonably good value for the standard deviation you need a sample size of at least 30.

I'll add that the t-tables are setup for the degrees of freedom in the sample. Calculating the average from the sample reduces the degrees of freedom by one, and calculating the standard deviation from the sample reduces the degrees of freedom by one again.