# Is it multiple comparisons?

Our study hypothesis is that metabolic syndrome (MetS) has impact on some outcome. There are at least five definitions of MetS in the literature. One of colleagues suggested using all five definitions of MetS in the study, because they have somewhat different characteristics. For me it's first just not elegant way to do the study. And looks like p-hunting. In addition, we don't intent to compare the definitions. I need to provide some valid arguments against using the five definitions for predictor variable. From a statistical view, isn't it a case of multiple comparisons?

• Closely related: stats.stackexchange.com/questions/468620/… Commented May 28 at 9:46
• If each of the five MetS definitions results in a simple MetS index value then you could calculate the first principal component (PC1) across the five index values and use PC1 as the predictor. If PC1 turns out significant, you know for sure that one or more of the indices were having an impact. The loadings on PC1 will tell you the importance of each index. Commented May 29 at 12:38
• Niels Holst, thank you. Could you please give an example from some research paper doing it? Commented May 30 at 14:04

I have written much of what I think of multiple comparison corrections here.

Some key points:

1. If you run several tests it's "multiple testing", but this doesn't necessarily imply that you must use a multiple testing correction. Multiple testing corrections have advantages and disadvantages, and ultimately if you provide precise p-values and state all that you have done, a reader can apply their favourite correction themselves. In the literature you find a hodgepodge of things where in many papers multiple testing corrections are applied over a certain set of tests and then some other tests are run that test something slightly different that are not involved.

2. The issue is really interpretation. I'd personally run all tests and then comment on the results of all of them. Let's say you run five tests and the smallest p-value you find is 0.03 and say all others are above 0.05 (let's say we want to est at 5%-level, which of course is not god-given). Now knowing Bonferroni, we know that running five tests, a minimum p-value of 0.03 doesn't tell us anything. If however all tests give you p-values around 0.03 and are testing "more or less the same thing", we would think that this may well mean something. If one p-value is 0.0001, we would also think that this probably means something even if all others are insignificant, but then we need to explain how the one test is different (also in meaning) from the others, so one would need to look at all five definitions and see how their differences can explain the result.

3. For sure it is wrong to pick the lowest p-value and interpret just that, but I think showing everything that was tested and interpreting with appropriate modesty what can be learnt from this (involving knowledge from multiple testing corrections such as saying "this one would still be significant under Bonferroni-Holm but not that one") should be fine.

We just need to get out of the mindset that a single p-value can give us all the story and then the case is closed. We can look at many things and learn, and looking at many things will also often teach us that we don't understand a situation as well as we might think looking only at a single one. (Also never look at p-values alone, always also look at effect sizes and standard errors and plot the data.)

• I am wondering how make journal reviewers think the same way as you've written here? I mean first of all "p-value mindset". Commented May 28 at 10:32
• @Ivan.S Very good question, to which unfortunately I don't have a good answer. :-( You can always be lucky though, not all reviewers are the same. Commented May 28 at 10:34

Yes, it's a case of multiple comparisons, if you want to evaluate all and decide which on to use afterwards. Depending on what kind of study you are doing that may be more or less of a problem. Certainly, for a study that's meant to be confirmatory (in the sense of having a clearly pre-specified null and alternative hypothesis that are meant to be addressed in a way that convince a research community), then it's awkward if your hypotheses aren't really clear and multiplicity would usually be seen as something one cannot just ignore. Naive adjustments (like, say, Bonferroni) will be rather non-ideal, as likely the test statistics would be highly correlated, but simplistic multiplicity adjustments would not exploit that.

One could pick one as the primary one to look at and then say that the others only provide a sensitivity check (i.e. does the definition used make much of a difference). One just has to live with the possibility of the primary definition not supporting the main research hypothesis, while other definitions would have. In those cases, one might for the sake of transparency end up with something like "Our pre-specified primary analysis did not ...., but some of our sensitivity analyses did." (instead of being able to ignore the pre-specified primary result) and also be willing to always write it the other way around, too, like "While our pre-specified primary analysis seems very convincing, some of our sensitivity analyses contradict it." (even if that undercuts the primary analysis a bit). However, this simply makes sense if one is not sure about the best definition to point out that the definition used matter.

• Thank you very much for such a clear and detailed answer! Will select primary definition and perform sensitivity analyses with the others. Commented May 28 at 9:55