normality test on small samples

Question

Premise: not very clever in statistics!

Data: I have quantitative data (two variables, A and B) on two small groups of subjects (both N=7); I'm going to perform a T-test to check differences about those groups. Before it, I run a normality test on data A and B of first group, using Matlab lillietest function. The test says A is normal (H0 holds), B is not (H0 rejected). Here are my data, followed by p values:

A       B    
0.000   0.125
1.500   0.125
2.375   1.125
2.375   0.125
5.625   0.250
4.250   0.000
0.750   0.000

p=0.37  p=0.008
H0=1    H0=0

I obtained similar results for second group. My conclusion is that I can perform t-test only for variable A, not for B.

Question 1) is my conclusion correct?

Question 2): given the very small dimension of sample size (n=7), how should I consider the normality test responses? Did I run a meaningful test? Is there a minimum sample size for it?

Answer 1

Elaboration on t.f's answer.

The normality test is a sneaky beast, because conceptually it works the other way round than a "normal" statistical test. Normally, you base your knowledge based on the rejection of the null. Here, the "desired" outcome ("proof" of normality) is the non-rejection. However, failure to reject is not the same as proving the null! The fact that cannot find an effect, does not mean it is not there.

With few samples, you will therefore never reject your hypothesis, so you are likely to falsely assume that your data is normal.

Conversely, if you have plenty of data, you will always reject your null, because no data in real world is perfectly normal. Consider human height - typically assumed, in biology, to have a normal distribution. In fact, it has been assumed to be normal for the past 150 years (ever since Galton). However, height has clear boundaries: it cannot be negative, it cannot be 100 meters. Therefore, it cannot be normally distributed.

You will find a more detailed discussion with numerous examples here.

So what can you do?

• Is there any reason to believe that your data is not normally distributed? Can you guess the distribution a priori? Typical examples may include bacterial growth or frequency of occurrence of an event.
• Use a q-q plot or a similar visual aid to make the decision.
• If forced by your thesis advisor to make a normality test, rather than focusing on the p-value alone, consider the effect size or calculate the skewness.
• Do you have similar data from other experiments? Can you use it to increase your sample size?

Answer 2

1. Your conclusion is correct, the normality assumption is required for t-test. However the t-test is fairly robust to violations of it. In any case you can use a non parametric for example the test Mann Whitney when you can guarantee the normality assumption.

2. You should, with such a small amount of observation the power of your test is small, and therefore the 'effect' is probably large if you detected it (meaning the data is very not normal). It usually the problem when you have thousand of observation you reject the hypothesis that the distribution is normal but it usually due to the fact that nothing is perfectly normal and you have a lot of power even when the 'effect' is weak.

Answer 3

1. No data are ever perfectly normally distributed, so the idea that the data have to be normal for the t-test to be applicable is wrong. If it were so, nobody could ever use the t-test.

2. For a good range of distributions, the t-test is asymptotically valid for large enough sample size due to the Central Limit Theorem, Slutsky Lemma, and the fact that the t-distribution with increasing degrees of freedom converges to the normal.

3. Obviously the given datasets are rather small, so it can be doubted that the sample size is large enough. The general question to ask however is not "are the data normally distributed?" (which they aren't anyway), but rather whether normality is violated in ways that will critically affect the behaviour of the t-test. This is mainly the case for large outliers and strong skewness. The 1.125 in sample B looks rather dodgy indeed. It would prompt me to use a Wilcoxon test here.

4. Normality tests can sometimes uncover critical violations of normality, so they give some information, however what they do is somewhat different from what is really required, as they may also detect harmless deviations, and occasionally miss a critical one. Unfortunately there is no reliable alternative as long as you don't feel perfectly confident to detect problems from looking at the data. There is also literature that suggests that preliminary normality testing should not be used, because it affects the theory behind the t-test (this also applies to visually detecting problems from the data, although this is rarely mentioned). For a review and discussion see https://arxiv.org/abs/1908.02218

5. Other model assumptions such as independence are often more problematic than normality.

Answer 4

There's one thing nobody talked about: just look at the given data with common sense.

Group B has 2 zeros, then small values and one 1.125. So - why? Typical result, typo (0.125 as the other ones), ....? This data looks so unstable that I would not trust it to be typical. Just try to imagine how a larger sample of this kind would look like - this becomes difficult (something larger than 1 once in a while, otherwise small values? Filling up the gap between small and large values?).

On the other hand, even though unstable, the values are clearly smaller than those of group A. So if you want to compare the groups (need to with a statistical test and cannot just show a dot plot of the measurements), then use a non-parametric test (Wilcoxon test, equivalent to Mann-Whitney-U) -> with such large differences lack of power is problem, so you do not even have to care about normality. This detailed testing of each and every step while losing track of the final aim is rather less scientific than omitting it ;)