Small sample sizes and the Kolmogorov-Smirnov test

I have split my data set into four categories but one of them only has four data points in it.

I have run K-S normality tests on the other three categories which are each normal and when the data set is not split into four it is also normal.

For this fourth category, do I assume it is normal, and run a parametric analysis, or assume it is non-normal and run a nonparametric analysis?

This is one of many calculations I have run on the same method using the K-S test for normality but would it be inconsistent for me to suddenly switch to a Shapiro-Wilk test for this one analysis?

• (i) the KS test is for completely specified distributions. It doesn't sound like you have that. (ii) Why are you testing normality? Dec 11, 2013 at 2:21
• The best "test" for a subset with 4 points is likely to be informal. If any of 4 points looks like an outlier with respect to the rest of the data, really watch out and think if that makes sense scientifically. If those 4 points look asymmetric, watch out. If you test it as you tested the other subsets and flag that you know the test is of limited worth, you may cover many of the possible reactions. If you don't test, be prepared for an argument about consistency of method. My own flag is that I never use K-S to test for normality, but look at quantile plots and transform or use a GLM, etc. Dec 11, 2013 at 9:36

1 Answer

The Kolmogorov-Smirnov (K-S) test can be used to compare other distributions, so if you have enough data in the fourth category, I think you should be able to compare its distribution to the distribution(s) of the data in the other categories. It seems unlikely that you will have enough power to reject the null with only four data points in the fourth category's distribution though. I doubt this would provide sufficient power for most NHSTs, but there may be exceptions.

I think you should probably know already if it's safe to assume your four data points in the fourth category were sampled randomly from a normal distribution. That is, if you don't know that it's safe to assume this for prior theoretical and methodological reasons, it probably isn't safe to assume. However, the degree and manner of deviation from a normal distribution is more important in most cases than the simple, binary outcome of a NHST of your distribution's normality. You may want to consider a related, popular question here on Cross Validated: Is normality testing 'essentially useless'? Here's a useful quote I found in an answer to that question:

As a rule of thumb (not a law of nature), inference about means is sensitive to skewness and inference about variances is sensitive to kurtosis.

This suggests you might at least wish to test the skewness or kurtosis separately, focus on whichever is more relevant to the subsequent analysis you have in mind, and pay attention to the value of the skewness or kurtosis statistic itself, not just the $p$ value of whatever NHST to which you might've felt obligated to cede inferential authority.

Nonparametric analyses do not assume non-normal distributions. Most also possess power and reliability that match equivalent parametric tests. Therefore you probably haven't got much to lose by just defaulting to the nonparametric alternative and skipping the normality test entirely. The main tradeoffs are usually computational complexity and the need to provide references due to methodological unconventionality, but all this necessitates are a modern computer and a little extra reading and writing, so it's probably generally advisable to just go for the nonparametric alternative. This depends on what analysis you have in mind though, so you may want to look into the drawbacks of the particular nonparametric alternative you have in mind for yourself.

Last, I don't see how you'd do any additional harm in switching to a Shapiro-Wilk (S-W) test, but if you want to compare its result to those of the K-S tests, I'd say you should also perform S-W tests on those distributions so you have some idea of how the method variance affects your test statistics within each sample distribution. Also, bear in mind that the S-W test is directly contraindicated by many of the answers to the aforementioned question...but I assume many of those objections would apply similarly to a K-S test that references a normal distribution.