Is it meaningful to test for normality with a very small sample size (e.g., n = 6)?

Question

I have a sample size of 6. In such a case, does it make sense to test for normality using the Kolmogorov-Smirnov test? I used SPSS. I have a very small sample size because it takes time to get each. If it doesn't make sense, how many samples is the lowest number which makes sense to test?

Note: I did some experiment related to the source code. The sample is time spent for coding in a version of software (version A) Actually, I have another sample size of 6 which is time spent for coding in another version of software (version B)

I would like to do hypothesis testing using one-sample t-test to test whether the time spent in the code version A is differ from the time spent in the code version B or not (This is my H1). The precondition of one-sample t-test is that the data to be tested have to be normally distributed. That is why I need to test for normality.

Answer 1

Yes.

All hypothesis tests have two salient properties: their size (or "significance level"), a number which is directly related to confidence and expected false positive rates, and their power, which expresses the chance of false negatives. When sample sizes are small and you continue to insist on a small size (high confidence), the power gets worse. This means that small-sample tests usually cannot detect small or moderate differences. But they are still meaningful.

The K-S test assesses whether the sample appears to have come from a Normal distribution. A sample of six values will have to look highly non-normal indeed to fail this test. But if it does, you can interpret this rejection of the null exactly as you would interpret it with higher sample sizes. On the other hand, if the test fails to reject the null hypothesis, that tells you little, due to the high false negative rate. In particular, it would be relatively risky to act as if the underlying distribution were Normal.

One more thing to watch out for here: some software uses approximations to compute p-values from the test statistics. Often these approximations work well for large sample sizes but act poorly for very small sample sizes. When this is the case, you cannot trust that the p-value has been correctly computed, which means you cannot be sure that the desired test size has been attained. For details, consult your software documentation.

Some advice: The KS test is substantially less powerful for testing normality than other tests specifically constructed for this purpose. The best of them is probably the Shapiro-Wilk test, but others commonly used and almost as powerful are the Shapiro-Francia and Anderson-Darling.

This plot displays the distribution of the Kolmogorov-Smirnov test statistic in 10,000 samples of six normally-distributed variates:

Based on 100,000 additional samples, the upper 95th percentile (which estimates the critical value for this statistic for a test of size $\alpha=5\%$) is 0.520. An example of a sample that passes this test is the dataset

0.000, 0.001, 0.002, 1.000, 1.001, 1000000

The test statistic is 0.5 (which is less than the critical value). Such a sample would be rejected using the other tests of normality.

Answer 2

As @whuber asked in the comments, a validation for my categorical NO. edit : with the shapiro test, as the one-sample ks test is in fact wrongly used. Whuber is correct: For correct use of the Kolmogorov-Smirnov test, you have to specify the distributional parameters and not extract them from the data. This is however what is done in statistical packages like SPSS for a one-sample KS-test.

You try to say something about the distribution, and you want to check if you can apply a t-test. So this test is done to confirm that the data does not depart from normality significantly enough to make the underlying assumptions of the analysis invalid. Hence, You are not interested in the type I-error, but in the type II error.

Now one has to define "significantly different" to be able to calculate the minimum n for acceptable power (say 0.8). With distributions, that's not straightforward to define. Hence, I didn't answer the question, as I can't give a sensible answer apart from the rule-of-thumb I use: n > 15 and n < 50. Based on what? Gut feeling basically, so I can't defend that choice apart from experience.

But I do know that with only 6 values your type II-error is bound to be almost 1, making your power close to 0. With 6 observations, the Shapiro test cannot distinguish between a normal, poisson, uniform or even exponential distribution. With a type II-error being almost 1, your test result is meaningless.

To illustrate normality testing with the shapiro-test :

shapiro.test(rnorm(6)) # test a the normal distribution
shapiro.test(rpois(6,4)) # test a poisson distribution
shapiro.test(runif(6,1,10)) # test a uniform distribution
shapiro.test(rexp(6,2)) # test a exponential distribution
shapiro.test(rlnorm(6)) # test a log-normal distribution

The only where about half of the values are smaller than 0.05, is the last one. Which is also the most extreme case.

---

if you want to find out what's the minimum n that gives you a power you like with the shapiro test, one can do a simulation like this :

results <- sapply(5:50,function(i){
  p.value <- replicate(100,{
    y <- rexp(i,2)
    shapiro.test(y)$p.value
  })
  pow <- sum(p.value < 0.05)/100
  c(i,pow)
})

which gives you a power analysis like this :

from which I conclude that you need roughly minimum 20 values to distinguish an exponential from a normal distribution in 80% of the cases.

code plot :

plot(lowess(results[2,]~results[1,],f=1/6),type="l",col="red",
    main="Power simulation for exponential distribution",
    xlab="n",
    ylab="power"
)