Suppose I do $1000$ normality tests, and set $\alpha = 0.05$. When testing for normality, should I correct for multiple comparisons? So if $p>\frac{0.05}{1000}$, I should keep the null hypothesis (think the data is normally distributed)? I am using the Shapiro-Wilks test.

  • 2
    $\begingroup$ Why are you performing these normality tests? $\endgroup$ – EdM Jun 21 '15 at 19:11
  • 1
    $\begingroup$ Whether one should use multiple comparisons when doing multiple tests depends on the purpose you're using it for. What are you testing for normality, and (as EdM also asks), why are you testing that? $\endgroup$ – Glen_b Jun 21 '15 at 23:36
  • $\begingroup$ I need to decide between parametric and non-parametric test for 1,000 variables. I know that normality tests do not actually answer should I pick parameteric or non-parametric test but it is best option I have (given the too large data to be manually checked). I think most variables are normally distributed. I would like to find the few non-normal ones, without a huge number of false positives. $\endgroup$ – nor Jun 22 '15 at 7:14
  • $\begingroup$ @nor check this thread, you may find it interesting: stats.stackexchange.com/questions/2492/… $\endgroup$ – Tim Jun 22 '15 at 18:15

The answer depends on what risks you are trying to minimize. In a typical situation with multiple testing you are trying to avoid false-positive findings, so you take into account the fact that tests at $\alpha = 0.05$ might give a false-positive result 5% of the time.

In your case, however, you want to identify variables that are non-normally distributed so that you will use non-parametric tests on those variables. If you think that identifying non-normal distributions is very important, it seems that you would want to minimize false-negative findings (that is, you don't want to miss variables that really are non-normal). In that case adjusting for multiple comparisons works against that goal. One might even argue that $\alpha = 0.05$ is not sufficiently stringent, if you believe that applying a parametric test to a variable without a normal distribution is really risky.

That said, you might be worrying too much about the normal distributions of the variables, depending on the specifics of your analysis. Some argue that testing for normal distributions of variables is often useless; you should read that question and its answers before you proceed. In linear regression, normal distributions of variables themselves are not required, although normal distributions of residual errors are needed for some types of significance tests. If some of your variables are categorical, they can't be normally distributed in any event. There's a nice example here of useful linear regression on variables that aren't normally distributed. The t-test can be robust against lack of normality, depending on the variable distributions. So evaluate carefully whether you should even be doing this testing.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.