Is correction for multiple testing necessary when almost all variables show p-values < 0,05?

Question

If you compare patients and controls with respect to many clinical variables, making a large number of statistical tests and finding that only few of the variables are significantly different, whereas most of the variables show no significant difference between patients and controls.

Then it is obvious that the obtained p-values should be corrected for multiple testing.

However, if most of the variables are significantly different and very few are the same for patients and controls, shall we correct? If yes, what kind of correction do you propose?

Answer 1

Without context I'd say you'd have to correct for multiple testing in both of the described settings. Moreover, you should plan any correction for multiple testing before seeing your results. So irregardless of finding a relatively large amount of 'significant' differences or finding a relatively small amount, you, as the researcher, have to consider in advance whether you aren't 'blowing up' your assumed alpha level by performing a shotgun-blast of hypothesis tests.

When you are doing a lot of hypothesis testing however the most important questions you might ask yourself are: "why am I doing all these anyway?" and "do I need an hypothesis test to see these are different?"

Answer 2

Yes, it generally is (agreeing with @IWS that you should decide this beforehand). Indeed, the most basic correction, Bonferroni, compares the p-values to $\alpha/q$ where $q$ is the number of multiple tests and $\alpha$ is the nominal level. So p-values between $\alpha$ and $\alpha/q$ would be redeclared as nonsignificant after this multiple testing correction.

Conversely, if none of the p-values is less than $\alpha$, you may as well skip the correction, because multiple testing procedures generally avoid spurious rejections by making rejections more difficult (see e.g. above). So if there are no rejections in the first place even without correction, there won't be any, a fortiori, after a correction.

Answer 3

IWS provides the most relevant fact about correcting for multiple comparisons: You should not decide to use a correction based on what you see in your results (e.g. how many of the comparisons are significant when uncorrected), but rather based on what you expect your actual alpha would be for an analysis relative to what alpha you're comfortable with. In general, if you have a couple specific comparisons of interest (a priori, not based on what looks interesting in the data), you're better off testing those separately as contrasts. Then you can compliment that hypothesis testing with a more general description of the patterns in the data provided by post hoc tests, correcting for multiple comparisons.

I want to add that sometimes the question is not about a difference in any particular outcome(s), but rather about a general pattern across a whole set of outcomes. From what you describe, it sounds like that may be the case here. If your research question is something like "Do patients differ from controls on X? How about on Y? And Z?" where you really care about the particulars of X, Y and Z, then comparing patients vs. controls on each of them makes sense (the analysis you've asked about). However, if you research question is more like "Do patients differ from controls? I measured them on X, Y, and Z." where you have a set of variables that you think are relevant to the difference between patients and controls, but each one isn't really interesting on its own, then you may be better off with a multivariate test that tests them all simultaneously. If group (patient vs. control) is your only predictor, then you can run this as a MANOVA, putting all of your outcome variables in at once and looking for differences between groups across all of them. Not only will this be a more direct test to answer your question (if your question is about global differences rather than particular outcomes), it will also generally be more powerful. It also solves your problem of multiple comparisons, of course, since you are able to conduct the whole analysis with one test.