# Is correction for multiple testing necessary when almost all variables show p-values < 0,05?

If you compare patients and controls with respect to many clinical variables, making a large number of statistical tests and finding that only few of the variables are significantly different, whereas most of the variables show no significant difference between patients and controls.

Then it is obvious that the obtained p-values should be corrected for multiple testing.

However, if most of the variables are significantly different and very few are the same for patients and controls, shall we correct? If yes, what kind of correction do you propose?

Without context I'd say you'd have to correct for multiple testing in both of the described settings. Moreover, you should plan any correction for multiple testing before seeing your results. So irregardless of finding a relatively large amount of 'significant' differences or finding a relatively small amount, you, as the researcher, have to consider in advance whether you aren't 'blowing up' your assumed alpha level by performing a shotgun-blast of hypothesis tests.

When you are doing a lot of hypothesis testing however the most important questions you might ask yourself are: "why am I doing all these anyway?" and "do I need an hypothesis test to see these are different?"

• The last paragraph of this answer is particularly valuable. Commented Jan 17, 2017 at 13:37
• If you do a large number of tests you might want to consider FDR (false discovery rate) as a criterion. Commented Jan 17, 2017 at 15:20

Yes, it generally is (agreeing with @IWS that you should decide this beforehand). Indeed, the most basic correction, Bonferroni, compares the p-values to $\alpha/q$ where $q$ is the number of multiple tests and $\alpha$ is the nominal level. So p-values between $\alpha$ and $\alpha/q$ would be redeclared as nonsignificant after this multiple testing correction.

Conversely, if none of the p-values is less than $\alpha$, you may as well skip the correction, because multiple testing procedures generally avoid spurious rejections by making rejections more difficult (see e.g. above). So if there are no rejections in the first place even without correction, there won't be any, a fortiori, after a correction.

• All right. Apparently that is not convenient from the app, so I hope I think of it when in front of my PC. If not I appreciate a reminder :-) Commented Jan 21, 2017 at 14:31

IWS provides the most relevant fact about correcting for multiple comparisons: You should not decide to use a correction based on what you see in your results (e.g. how many of the comparisons are significant when uncorrected), but rather based on what you expect your actual alpha would be for an analysis relative to what alpha you're comfortable with. In general, if you have a couple specific comparisons of interest (a priori, not based on what looks interesting in the data), you're better off testing those separately as contrasts. Then you can compliment that hypothesis testing with a more general description of the patterns in the data provided by post hoc tests, correcting for multiple comparisons.