# Is the following approach a reasonable way of defining $p$-value thresholds?

I have many small sets of data (each sample containing 8 to 64 individual points) which I would like to test for normality. I want to do this with two main tests:

• Shapiro-Wilk
• Kolmogorov–Smirnov

However, both of these will produce $$p$$-values. Now I want to avoid the classic trope of using the $$p<0.05$$ threshold. As from what I have read (and as other users on CV have taught me) is that this is essentially an arbitrary "convention" which has emerged because many researchers do not know how to correctly use and interpret $$p$$-values for hypothesis tests.

As I understand it, one should define a $$p$$-value threshold on a case-by-case basis and should be motivated by scientific reasoning.

In my case I am unable to measure or identify non-normal sources which can contaminate my small sample sizes. So in my attempt to construct reasonable $$p$$-value thresholds I tried the following simulation:

1. generate simulated normal data sets for small sample sizes, for example 16 data points per sample;
2. Perform a normality test on the generated data set, and produce both a test statistic and a $$p$$-value;
3. repeat many times (e.g. 10,000) and look at the distribution of test statistics (which is normal) and then plot $$p$$-value as a function of test statistic:

I would then use some measure of the range of the test statistic distribution to define which $$p$$-values I use as a threshold test. For example in the Shapiro-Wilk test the distribution of test statistics ranges from $$0.85$$ to $$0.99$$ which would correspond to a $$p$$-value range of $$\sim0.02$$ to $$1.0$$

My rationale is that given I have no knowledge of non-normal contamination, the best I can do is to consider the fact that my data samples are small and use that as a measure.

Is this a reasonable approach?

In response to a comment:

In perfect conditions my data will be normally distributed -- I know this from the physics of the situation. However the environment the experiment is in can randomly fluctuate, sometimes in a way which is easily identifiable, and sometimes more subtly. By testing subsets of data for normality I can be sure (or as sure as I can be) that a particular data set is the phenomena I want to measure and not a convolution or mixture of phenomena and environmental effects.

Of course the ideal scenario would be to understand the behaviour of the non-normal influence and simulate. But in reality this is extremely difficult to quantify, hence my attempted solution.

• Although you refer to "scientific reasoning," you don't supply much information that could help there. Could you explain why you want to test these datasets for normality? What do you hope that might accomplish?
– whuber
Commented Oct 17, 2020 at 15:35
• @whuber A fair comments I have added an addendum to try and answer the points you raise. Essentially I know the true phenomena I am interested in will be normally distributed, from the physics of the problem. However the non-normal contamination are too complex for me to quantify and simulate in any meaningful way. Commented Oct 17, 2020 at 15:53
• Thank you for the explanation. Your approach was popular early in the 19th century until people realized that such tests have exceptionally little power: indeed, they have no power at all to detect independent Normally distributed noise, because the resulting distribution will still be Normal.
– whuber
Commented Oct 17, 2020 at 17:03
• Interesting, what would you suggest as an alternative? I use some estimates of scale such as the MAD, $S_{n}$, or $Q_{n}$ estimators to eliminate outliers in a given sample. And then I wanted to test the remainder for normality, if it fails a normality test I reject the entire sample, if it passes I keep it and use it in the analysis. The question is: what to do with my damn $p$-values?! Commented Oct 17, 2020 at 17:21

• The $p$-value shouldn't be chosen arbitrarily though, it should be chosen with scientific reasoning, and simulation. Commented Oct 17, 2020 at 17:29
• But if you don't know anything about your data you may accept or reject your hypothesis out of ignorance. $0.05$ is itself arbitrary, just because it's a convention doesn't mean it's good or effective. To me that just says a lot of people don't understand $p$-values. Commented Oct 17, 2020 at 17:39