3
$\begingroup$

While reading papers on my research topic I came across multiple paper formulating sentences similar to this:

Because of the hypothesis-generating nature of these analyses, no correction for multiple comparisons was performed.

I am a bit confused, what makes an analysis hypothesis-generating? If I run tests without having a hypothesis? And why does this mean, that I don't have to correct for multiple comparisons?

Could I perform a hypothesis testing analysis on data, and then on top of it run hypothesis-generating analysis?

$\endgroup$

1 Answer 1

7
$\begingroup$

This is a complex issue. I have written something about multiple testing here, and the baseline is that corrections for multiple comparisons have advantages and disadvantages and the issue is not "black-and-white" as if in some situations you'd "have to" do this and not in others. Particularly, the answer to this cannot be prescribed by any theory, but is rather of a pragmatic or even philosophical nature. Ultimately it depends on how results are interpreted. Surely, by running many tests without multiple testing correction there is a high probability to find false rejections of null hypotheses, much higher than the nominal level, and this is dangerous and should be avoided if you intend to assign a strong meaning to a significant result. What the cited authors apparently mean by "hypothesis generation" is that if they find something significant, the interpretation is not that "there is strong evidence that something meaningful is going on", but rather that "it may be worthwhile to have a deeper look into this on new data". In such a case, if the authors run, say, 20 tests, they'd be happy to have, say, 4 significant outcomes to investigate further (well taking into account that 2, 3, or even all 4 of them could be meaningless), so what this does is that it narrows the scope for future research, which may be fine, even if these results don't say anything reliable about any specific one of the outcomes, which would require new data and more research. (Conversely this means that if you want to make reliable statements about any of these hypotheses, don't do it like this.)

When reading such work, I'd advise to keep a skeptical attitude though, because occasionally one finds such remarks made by authors in the methodological part of their papers, and then in their conclusions they treat the significant results as if they were indeed really meaningful anyway, and no further research on new data is really intended. In this case talking about "hypothesis generating" research is just a cheap excuse to get around multiple testing corrections. (Personally I doubt by the way that testing lots of hypotheses and then investing those further that more or less randomly come out significant is a very convincing research program, at least in absence of a proper subject matter justification of what exactly to investigate; however I occasionally comment on non-corrected significances in exploratory data analysis, for example looking at a correlation matrix, as "things to have an eye on" or similar.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.