# Do statistical tests require a normal distribution in the sample or in the population?

Certaint tests, such as the t-test, require a normal distribution. Yet I have never quite understood what exactly has to be normally distributed.

Let's use age as an example. Age is rarely normally distributed in the population:

Yet, in a specific sample age might very well appear normally distributed. Here is an example from a recent study:

age.in.months <- c(156, 127, 160, 183, 137, 158, 132, 131, 126,
199, 138, 141, 126, 154, 130, 148, 163, 158, 139, 101, 176, 155,
115, 177, 150, 123, 124, 151, 177, 121, 170, 137, 162, 136, 161,
146, 141, 152, 154, 170, 137, 137, 160, 142, 126, 150, 129, 144,
175, 134, 117, 158, 157, 157, 127, 175, 128, 130, 137, 148, 130,
194, 151, 133, 184, 132, 125, 160, 175, 137, 115, 191, 171, 147,
162, 136, 134, 178, 165, 173, 133, 117, 138, 145, 141, 154)


Shapiro-Wilk normality test
W = 0.97908, p-value = 0.1769

Shapiro-Francia normality test
W = 0.98057, p-value = 0.1913

Anderson-Darling normality test
A = 0.67431, p-value = 0.07562

Cramer-von Mises normality test
W = 0.11377, p-value = 0.07141


In this sample, age appears to be borderline normally distributed.

With most data samples, we do not know how that trait is distributed in the population and have only the sample distribution to go by. But with some traits, such as age, we either know or have a very strong indication as to the distribution of that trait in the population.

For these traits, where we know the distribution in the population, which distribution – the one in the population or the one in the sample – has to be normally distributed for us to use tests that require normality?

As I see it, the (almost) normal distribution of age in the example sample is merely an artifact of the sampling. We wanted participants to be within a certain age range, and the limits we have set may have caused a drop in participant numbers close to the edges of that range.

If I understand it correctly, the Wikipedia article on parametric statistics states that it is the population in which the trait must be normally distributed: "Parametric statistics is a branch of statistics which assumes that sample data comes from a population that follows a probability distribution based on a fixed set of parameters. ... we assume all ... test scores are random samples from a normal distribution ..."

And in the slides from his lecture on inferential statistics, my professor states that: "To answer the question, whether the means in both samples are the same, we have to go beyond the current samples and look at the situation in the population that we want to study. To infer the situation in the population from the situation in the sample data, we assume that the characteristic that we want to study is normally distributed in the population. This modelling is not theoretical but must be established empirically. We therefore check whether the sampled data can be considered as coming from a normal distribution." (my translation) This, to me, would imply that if we know that the trait is not normally distributed in the population, we must not use parametric tests, no matter the sample distribution.

• Since the t-test is being done on the samples, those will need to be normally distributed. And for linear models (regression, ANOVA), it is the residuals that need to be normally distributed, not the raw values. – mkt - Reinstate Monica Jul 6 '17 at 7:41
• Thank you, @mkt. Can you, or someone, explain why the distribution in the population is irrelevant? – user167929 Jul 6 '17 at 8:20
• It is not at all irrelevant. The p value of a one sample t-test is precise if the sample values were drawn independently from the same normal distribution (= the "true" distribution of the "population"). In a normal linear model setting, the t-tests associated with the coefficients are exact if all assumptions hold and the error terms are drawn independently from the same normal distribution (the "true" distribution of the "population"). Typically, the distributional assumptions refer to distributions in the population, not in the sample. But we can only check them based on the sample. – Michael M Jul 6 '17 at 9:24
• To add to what @MichaelM said: you would expect your sample to have the same distribution as the population. If it isn't, either your sample is very small or something went wrong somewhere... – mkt - Reinstate Monica Jul 6 '17 at 9:44
• If eg age itself is not normaliy distributed, but you take the average age of the people in tour sample, then this average age must be normality distributed. Under reasonable assumptions thus Will be the case by the central limit theorem – user83346 Jul 6 '17 at 14:33

A random sample $X_1,X_2,\dots, X_n$ is understood as a collection of independent observations of one (population) variable $X$ which all share the same distribution $F$ (i.e. $N(\mu,\sigma^2)$).

This assumption may be quite intuitive ("hey it's a random sample, why should the distributions differ?!") and is in fact needed to draw more general conclusions from the sample level to the population level.

Indeed, if the sample $X_1,\dots, X_n$ wouldn't share the same distribution as $X$, how would you ever be able to estimate the whole distribution of $X$ with the help of $X_1,X_2,\dots, X_n$?

For example, a histogram would then only be useful to describe the distribution of the given sample, but could not be used to draw any conclusions about the distribution on the population level.

This is not what you want. Hence it's naturally to assume that $X_1,X_2,\dots, X_n$ are a sample of $X$ and as such follow the same distribution as $X$.

However, as mkt already pointed out, the special case of a $t$-Test could be interepreted that you are per se not interested in the whole distribution of $X$, but only interested in the population mean $E(X)$.

Hence for the $t$-Test it would indeed suffice to assume that $X_1, X_2, \dots, X_n$ are independent and each of them respects a $N(\mu,\sigma^2)$ distribution. Additionally we would only have to assume that $\mu = E(X)$, i.e. the distribution of $X$ could indeed be different from the distribution of $X_1,X_2, \dots, X_n$ and we only assume that the distribution of $X$ is such that $X$ and $X_1, X_2, \dots, X_n$ share the same mean. (But as pointed out above: by definition, this story would contradict the fact that $X_1,X_2,\dots, X_n$ is a random sample of $X$!)

To be more concrete: in the $t$-Test, the assumption that $X_1, X_2, \dots, X_n$ are i.i.d. $N(\mu,\sigma^2)$ is crucial (while in order to draw conclusions from the sample to the population, for the population variable $X$ we only would have to assume that $E(X)=\mu$, which is much weaker than assuming a whole distribution.). Indeed, if the sample is not normal distributed, the distribution of the test-statistic $$t= \sqrt{n}\frac{\overline{X}-\mu_0}{s}$$ would not follow a $t(n-1)$ distribution (given the hypothesis $\mu = E(X) = \mu_0$ is true) and it would be quite hard to derive the exact distribution of the test statistic given the sample.

However, staying with your example: you already concluded graphically that for any given nation the variable $X=age$ is not normal distributed. And as a result, if you draw a random sample from the whole population, the distribution of $X_1,X_2,\dots, X_n$ can never be normal distributed by the definition of a random sample.

As a matter of fact: $X=age$ can by itself NEVER be normal distributed since its range is bounded which contradicts already a normal distribution.

You may, like you have done, still test for it and a test may in fact let you conclude that the sample is normal distributed, but this conclusion will be clearly the result of an error you are doing with statistical testing:

There are two types of errors you can do in statistical testing. Suppose you assume that the population distribution is given by $F$ (for example normal distribution).

1. Type-A Error: The sample was a bad draw from $F$. The distribution of the sample is actually $F$, but the observations from the sample was so bad that the test rejected it.
2. Type-B Error: The underlying distribution is in fact different from $F$ but the sample does not contain enough evidence to reject the hypothesis that the distribution was $F$.

While testing you can control Type-A error (it's the level of significance) but typically you can not control the Type-B error (you would have to know the exact distribution for it.)

In your example you are clearly doing a Type-B error.

Coming back to the $t$-test:

In your example, $X$ as well as the sample $X_1,X2, \dots, X_n$ are not normal distributed. Hence the $t$-statistic

$$t= \sqrt{n}\frac{\overline{X}-\mu_0}{s}$$ will not be $t(n-1)$-distributed.

However if we are willing to assume $X_1,X2, \dots, X_n$ share the same distribution $F$ (whatever it may look like) with $E(X_i^2)<\infty$, we then may apply a central limit theorem and as a result may conclude that $t$ is approximately $N(0,1)$ distributed - regardless of the the underlying distribution of the sample/population.

This is the feasible version of the $t$-test, since the distribution of $X$ (and hence of each of the sample variables) will in practice never be exact normal distributed.

Of course, we then have to face an approximation error when calculating $p$-values but this error will be negligible provided the sample size is large enough (a rule of thumb is: $n>30$). On the other hand, for small sample sizes it will better to perform a nonparametric test.

• This answer is a little hard to read just once, because it winds up stating almost of the opposite of what it so forcibly asserts at the beginning. (It begins by saying normality is "crucial" and then says it's ok to relax that assumption when $n\gt 30$.) If we accept this as the evolution of an analysis, that's fine--provided we forget the beginning. But the rule of thumb is a poor one, with little justification: see, for instance, my comment to the question itself for why $n=5$ can work and see stats.stackexchange.com/questions/69898 for a real case where $n=10,000$ does not suffice. – whuber Jul 6 '17 at 14:46