I have many small sets of data (each sample containing 8 to 64 individual points) which I would like to test for normality. I want to do this with two main tests:

  • Shapiro-Wilk
  • Kolmogorov–Smirnov

However, both of these will produce $p$-values. Now I want to avoid the classic trope of using the $p<0.05$ threshold. As from what I have read (and as other users on CV have taught me) is that this is essentially an arbitrary "convention" which has emerged because many researchers do not know how to correctly use and interpret $p$-values for hypothesis tests.

As I understand it, one should define a $p$-value threshold on a case-by-case basis and should be motivated by scientific reasoning.

In my case I am unable to measure or identify non-normal sources which can contaminate my small sample sizes. So in my attempt to construct reasonable $p$-value thresholds I tried the following simulation:

  1. generate simulated normal data sets for small sample sizes, for example 16 data points per sample;
  2. Perform a normality test on the generated data set, and produce both a test statistic and a $p$-value;
  3. repeat many times (e.g. 10,000) and look at the distribution of test statistics (which is normal) and then plot $p$-value as a function of test statistic: enter image description here

I would then use some measure of the range of the test statistic distribution to define which $p$-values I use as a threshold test. For example in the Shapiro-Wilk test the distribution of test statistics ranges from $0.85$ to $0.99$ which would correspond to a $p$-value range of $\sim0.02$ to $1.0$

My rationale is that given I have no knowledge of non-normal contamination, the best I can do is to consider the fact that my data samples are small and use that as a measure.

Is this a reasonable approach?

In response to a comment:

In perfect conditions my data will be normally distributed -- I know this from the physics of the situation. However the environment the experiment is in can randomly fluctuate, sometimes in a way which is easily identifiable, and sometimes more subtly. By testing subsets of data for normality I can be sure (or as sure as I can be) that a particular data set is the phenomena I want to measure and not a convolution or mixture of phenomena and environmental effects.

Of course the ideal scenario would be to understand the behaviour of the non-normal influence and simulate. But in reality this is extremely difficult to quantify, hence my attempted solution.

  • 1
    $\begingroup$ Although you refer to "scientific reasoning," you don't supply much information that could help there. Could you explain why you want to test these datasets for normality? What do you hope that might accomplish? $\endgroup$
    – whuber
    Commented Oct 17, 2020 at 15:35
  • $\begingroup$ @whuber A fair comments I have added an addendum to try and answer the points you raise. Essentially I know the true phenomena I am interested in will be normally distributed, from the physics of the problem. However the non-normal contamination are too complex for me to quantify and simulate in any meaningful way. $\endgroup$
    – user27119
    Commented Oct 17, 2020 at 15:53
  • 2
    $\begingroup$ Thank you for the explanation. Your approach was popular early in the 19th century until people realized that such tests have exceptionally little power: indeed, they have no power at all to detect independent Normally distributed noise, because the resulting distribution will still be Normal. $\endgroup$
    – whuber
    Commented Oct 17, 2020 at 17:03
  • $\begingroup$ Interesting, what would you suggest as an alternative? I use some estimates of scale such as the MAD, $S_{n}$, or $Q_{n}$ estimators to eliminate outliers in a given sample. And then I wanted to test the remainder for normality, if it fails a normality test I reject the entire sample, if it passes I keep it and use it in the analysis. The question is: what to do with my damn $p$-values?! $\endgroup$
    – user27119
    Commented Oct 17, 2020 at 17:21

1 Answer 1


As mentioned in previous comments, these tests are very sensetive, and might be problematic to blindly rely on. Why don't you just visualize the data with a histogram or a qq-plot, and take it from there? My experience is that this might even be better than a test, since tests will always lead to a loss of information.

Regarding choosing the "correct" p-value, my understanding is that one of the reason to keep to that threshold is to counteract arbitary thresholds and cheating. For example, if the p-value is chosen arbitrary from test to test, it leaves room for researchers to choose p-value to confirm their hypotheses.

Plus, if you have many small samples which you want to test, you'll need to adjust for massignificance.

  • $\begingroup$ The $p$-value shouldn't be chosen arbitrarily though, it should be chosen with scientific reasoning, and simulation. $\endgroup$
    – user27119
    Commented Oct 17, 2020 at 17:29
  • $\begingroup$ My bad, I don't mean arbitrary. But the point still holds. Keeping to the convention of 0.05 minimizes the risk of academic dishonesty. Like choosing hypotheses after you observe the data. $\endgroup$
    – awend
    Commented Oct 17, 2020 at 17:32
  • $\begingroup$ But if you don't know anything about your data you may accept or reject your hypothesis out of ignorance. $0.05$ is itself arbitrary, just because it's a convention doesn't mean it's good or effective. To me that just says a lot of people don't understand $p$-values. $\endgroup$
    – user27119
    Commented Oct 17, 2020 at 17:39
  • $\begingroup$ I think that a general discussion about the p-value could be a post of itself, however I think most that can be said about it have already been said. In your case, for testing normality, I think you should be doing fine with some visualisation. Mass testing your small samples will just lead to more problems, regarding p-values. Why do you need to check your normality assumption with a hypothesis test? $\endgroup$
    – awend
    Commented Oct 17, 2020 at 17:52
  • $\begingroup$ A visual inspection works fine if you have sufficient data points to, resolve the distribution. I am dealing with small (8, 16, 32) sample sizes. A histogram can look like anything. A hypothesis test is necessary exactly because I have a small sample size. $\endgroup$
    – user27119
    Commented Oct 17, 2020 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.