# Multiple comparisons with normality tests

Suppose I do $1000$ normality tests, and set $\alpha = 0.05$. When testing for normality, should I correct for multiple comparisons? So if $p>\frac{0.05}{1000}$, I should keep the null hypothesis (think the data is normally distributed)? I am using the Shapiro-Wilks test.

• Why are you performing these normality tests? – EdM Jun 21 '15 at 19:11
• Whether one should use multiple comparisons when doing multiple tests depends on the purpose you're using it for. What are you testing for normality, and (as EdM also asks), why are you testing that? – Glen_b Jun 21 '15 at 23:36
• I need to decide between parametric and non-parametric test for 1,000 variables. I know that normality tests do not actually answer should I pick parameteric or non-parametric test but it is best option I have (given the too large data to be manually checked). I think most variables are normally distributed. I would like to find the few non-normal ones, without a huge number of false positives. – nor Jun 22 '15 at 7:14
• @nor check this thread, you may find it interesting: stats.stackexchange.com/questions/2492/… – Tim Jun 22 '15 at 18:15

The answer depends on what risks you are trying to minimize. In a typical situation with multiple testing you are trying to avoid false-positive findings, so you take into account the fact that tests at $\alpha = 0.05$ might give a false-positive result 5% of the time.
In your case, however, you want to identify variables that are non-normally distributed so that you will use non-parametric tests on those variables. If you think that identifying non-normal distributions is very important, it seems that you would want to minimize false-negative findings (that is, you don't want to miss variables that really are non-normal). In that case adjusting for multiple comparisons works against that goal. One might even argue that $\alpha = 0.05$ is not sufficiently stringent, if you believe that applying a parametric test to a variable without a normal distribution is really risky.