Is it meaningful to test for normality with a very small sample size (e.g., n = 6)?

I have a sample size of 6. In such a case, does it make sense to test for normality using the Kolmogorov-Smirnov test? I used SPSS. I have a very small sample size because it takes time to get each. If it doesn't make sense, how many samples is the lowest number which makes sense to test?

Note: I did some experiment related to the source code. The sample is time spent for coding in a version of software (version A) Actually, I have another sample size of 6 which is time spent for coding in another version of software (version B)

I would like to do hypothesis testing using one-sample t-test to test whether the time spent in the code version A is differ from the time spent in the code version B or not (This is my H1). The precondition of one-sample t-test is that the data to be tested have to be normally distributed. That is why I need to test for normality.

-
No............. –  Joris Meys Aug 8 '11 at 11:10
I, for one, have difficulty imagining a context in which n=6 and normality would be an hypothesis worth testing. I fear this is a case of an inexperienced user doing multiple hypothesis testing (run a regression then test for normality of residuals) and that we are addressing the symptoms but ignoring the skeletons in the closet, so to speak. –  user603 Aug 8 '11 at 14:45
User603 (re your first comment): I would like to point out that @Joris has not supplied an answer, nor is his comment accompanied with any justification whatsoever. If an emphatic "no" is a valid general answer to this question, let's see it written down as such, with a supporting argument, so it can be evaluated up and down by the community. –  whuber Aug 8 '11 at 15:03
@whuber : I added an argument for the emphatic "no". –  Joris Meys Aug 8 '11 at 15:46
@Joris Thank you! That is helpful and illuminating. –  whuber Aug 8 '11 at 16:00

Yes.

All hypothesis tests have two salient properties: their size (or "significance level"), a number which is directly related to confidence and expected false positive rates, and their power, which expresses the chance of false negatives. When sample sizes are small and you continue to insist on a small size (high confidence), the power gets worse. This means that small-sample tests usually cannot detect small or moderate differences. But they are still meaningful.

The K-S test assesses whether the sample appears to have come from a Normal distribution. A sample of six values will have to look highly non-normal indeed to fail this test. But if it does, you can interpret this rejection of the null exactly as you would interpret it with higher sample sizes. On the other hand, if the test fails to reject the null hypothesis, that tells you little, due to the high false negative rate. In particular, it would be relatively risky to act as if the underlying distribution were Normal.

One more thing to watch out for here: some software uses approximations to compute p-values from the test statistics. Often these approximations work well for large sample sizes but act poorly for very small sample sizes. When this is the case, you cannot trust that the p-value has been correctly computed, which means you cannot be sure that the desired test size has been attained. For details, consult your software documentation.

Some advice: The KS test is substantially less powerful for testing normality than other tests specifically constructed for this purpose. The best of them is probably the Shapiro-Wilk test, but others commonly used and almost as powerful are the Shapiro-Francia and Anderson-Darling.

This plot displays the distribution of the Kolmogorov-Smirnov test statistic in 10,000 samples of six normally-distributed variates:

Based on 100,000 additional samples, the upper 95th percentile (which estimates the critical value for this statistic for a test of size $\alpha=5\%$) is 0.520. An example of a sample that passes this test is the dataset

0.000, 0.001, 0.002, 1.000, 1.001, 1000000


The test statistic is 0.5 (which is less than the critical value). Such a sample would be rejected using the other tests of normality.

-
+1, very good answer. –  mpiktas Aug 8 '11 at 13:13
I think any distribution that gives a sig. result with N = 6 will be so non normal that it will pass the IOTT with flying colors - that's the interocular trauma test. It hits you between the eyes. –  Peter Flom Aug 8 '11 at 13:24
@Peter If you were to rephrase this comment, it would be correct. After all, many $N=6$ samples from a normal distribution will look perfectly normal, so clearly "any" is too strong a quantifier. What you meant to say is that there's a good chance that a random sample with $N=6$ will be clearly non-normal when plotted in a reasonable way (e.g., probability plot) but will not be rejected by this test. –  whuber Aug 8 '11 at 13:27
@Peter Good! A KS test for normality has rejected a uniform sample. That's what one hopes. –  whuber Aug 8 '11 at 14:12
set.seed(140);x=rnorm(6);ks.test(x,pnorm) produces p-value = 0.0003255. Of course I had to try it with 140 seeds before I found this... –  Spacedman Aug 9 '11 at 15:22
show 1 more comment

As @whuber asked in the comments, a validation for my categorical NO. edit : with the shapiro test, as the one-sample ks test is in fact wrongly used. Whuber is correct: For correct use of the Kolmogorov-Smirnov test, you have to specify the distributional parameters and not extract them from the data. This is however what is done in statistical packages like SPSS for a one-sample KS-test.

You try to say something about the distribution, and you want to check if you can apply a t-test. So this test is done to confirm that the data does not depart from normality significantly enough to make the underlying assumptions of the analysis invalid. Hence, You are not interested in the type I-error, but in the type II error.

Now one has to define "significantly different" to be able to calculate the minimum n for acceptable power (say 0.8). With distributions, that's not straightforward to define. Hence, I didn't answer the question, as I can't give a sensible answer apart from the rule-of-thumb I use: n > 15 and n < 50. Based on what? Gut feeling basically, so I can't defend that choice apart from experience.

But I do know that with only 6 values your type II-error is bound to be almost 1, making your power close to 0. With 6 observations, the Shapiro test cannot distinguish between a normal, poisson, uniform or even exponential distribution. With a type II-error being almost 1, your test result is meaningless.

To illustrate normality testing with the shapiro-test :

shapiro.test(rnorm(6)) # test a the normal distribution
shapiro.test(rpois(6,4)) # test a poisson distribution
shapiro.test(runif(6,1,10)) # test a uniform distribution
shapiro.test(rexp(6,2)) # test a exponential distribution
shapiro.test(rlnorm(6)) # test a log-normal distribution


The only where about half of the values are smaller than 0.05, is the last one. Which is also the most extreme case.

if you want to find out what's the minimum n that gives you a power you like with the shapiro test, one can do a simulation like this :

results <- sapply(5:50,function(i){
p.value <- replicate(100,{
y <- rexp(i,2)
shapiro.test(y)$p.value }) pow <- sum(p.value < 0.05)/100 c(i,pow) })  which gives you a power analysis like this : from which I conclude that you need roughly minimum 20 values to distinguish an exponential from a normal distribution in 80% of the cases. code plot : plot(lowess(results[2,]~results[1,],f=1/6),type="l",col="red", main="Power simulation for exponential distribution", xlab="n", ylab="power" )  - @whuber : regarding the logic of hypothesis testing on its head : in which case are you interested in the alternative hypothesis? In all applications of these tests I've seen, people are interested in the confirmation of the null : my data do not differ significantly from a normal distribution. Which is why I emphasize the type II-error. – Joris Meys Aug 8 '11 at 16:39 See my comments to the OP concerning groundwater monitoring. Typically people are interested in rejecting one or both of two default assumptions: normality and lognormality. Because this is done under strict regulatory supervision, eyeballing a probability plot (which is a powerful tool for experienced IOTT practitioners like @Peter Flom) does not suffice: formal tests are needed. A similar application occurs in human health risk assessment; US EPA guidance documents specifically contemplate tests with$n$as low as$5$. See epa.gov/oswer/riskassessment/pdf/ucl.pdf, e.g.. – whuber Aug 8 '11 at 16:53 To get back to the title: is it meaningful to test for normality with small sample sizes? In some cases it is, especially when testing against strongly skewed alternatives. (SW has 80% power at$n=8$against an LN(1,2) alternative, e.g.) Low power against many alternatives when$n\$ is small is something normality tests share, to one degree or another, with any hypothesis test. That does not preclude its use. Thus, an unqualified "no" is, to put it mildly, unfair to the test. More generally, it suggests we shouldn't ever use hypothesis tests on "small" samples ever. That sounds too Draconian. –  whuber Aug 8 '11 at 20:29
@whuber : We'll have to agree to differ. I'm not completely a fan of EPA (and definitely not of FDA) guidelines. I've seen this abused once too often to still believe in its usefulness. Chance is a weird thing, and with only 6 cases highly unpredictable. I don't believe you can say anything about a complex function like a PDF based on only 6 observations. YMMV –  Joris Meys Aug 8 '11 at 20:54
@ImAlso The t-test can tolerate a lot of non-normality if it's fairly symmetric, but it can't tolerate too much asymmetry. (Indeed, a skewness test for normality might actually be a better option in the O.P. than the K-S test, for just this reason.) This points out one of the biggest differences between goodness of fit tests and other hypothesis tests: there is a huge space of possible alternatives and the GoF tests tend to be good against certain of them but not against others. You can't make them work well against all alternatives. –  whuber Aug 10 '11 at 22:30