1
$\begingroup$

and thanks for taking the time to look at this thread!

I'm massively stumped about exactly when it's 'correct' to be correcting for multiple comparisons by adjusting the threshold p-value. I know in post-hoc tests after an omnibus test, it's absolutely the 'done thing', and I can see why: by reducing the threshold p-value, you reduce the chance of making a Type-I error, trying to make it so you still only have a 5% chance (when p≤0.05) of getting a false-positive, thus adjusting for the number of tests you run. But why is this logic not extended to all tests in an article?


My research (and specific problem)

To give some background of the design I'm using (and hence why I'm stumped): we have 8 questions, all which are asked for 4 'categories, and within each 'category', we have 2 stimuli which participants all rate simultaneously. The questions are entirely different, and we have different hypotheses for what should happen with each, so the scores can't be combined in any way. Thus, we're running Wilcoxon's Signed Ranks test (the data is, unfortunately, non-parametric) between each set of stimuli for all categories and questions: thus, we have 32 tests.

Due to the sheer volume of the tests we're running, it seems highly likely that at least one will be a 'false-positive' when p≤0.05. That suggests to me that the p-value should be reduced to control for these multiple comparisons.

But then, each tests a functionally discrete hypothesis, so why should we have to? You wouldn't do that if you ran several ANOVA-type omnibus tests, or if you ran any number of different tests on the same results (e.g., regression and factor analysis, as a relatively weak example).

And to add further confusion, this is one of four studies of relatively similar design. Surely, if we want to reduce finding any 'false-positives', we should control the p-value across all tests conducted before writing up the article?

So, as you can see, I'm pretty stumped. I realise controlling for all comparisons in a paper seems ridiculous, and must be wrong, but it follows all the logic I've seen so far about why we correct for multiple comparisons in the first place. But I'm fully prepared to accept I'm an idiot!

$\endgroup$
1
$\begingroup$

This is a difficult topic and it is hard to know what is right. If I were to control the type 1 error rate in each paper, why can I suddenly claim something just by splitting a paper into two papers? See also Berger, Vance W. "On the generation and ownership of alpha in medical studies." Controlled Clinical Trials 25.6 (2004): 613-619.

In medical trials a convention now seems to be to have a set of primary and key secondary endpoints (or comparisons) across which the type 1 error rate is controlled, while this is not done for other secondary/tertiary/exploratory endpoints/comparisons (which have a lot less credibility to most). However that is really just a convention in one field.

$\endgroup$
1
$\begingroup$

That's a reasonable question, so I don't think you're an idiot.

First of all, the problem of multiple comparisons is not specific to tests following ANOVA. It applies to multiple testing scenarios in general. So it's not necessarily "ridiculous" or "wrong" to apply correction to multiple ANOVAs or to every comparison in a paper. Any time you give yourself multiple opportunities to find significance, you inflate the probability of "getting lucky."

It's hard to give specific advice without knowing your specific situation--what the hypotheses are, whether some hypotheses are more important than others, to what extent the hypotheses are correlated, what the goal of the study is (e.g., to get published, to make some decision about a follow-up study), etc.

How did you end up with 32 completely unrelated hypotheses in the same study in the first place? You mentioned that you plan to write up an "article," but maybe you're in the very early stages of investigating a broad line of research and haven't figured out which specific hypotheses are the most interesting or the most promising yet. In that case, maybe you should use a fairly liberal approach to error control (such as FDR) to identify which hypotheses may be worth pursuing further. You might want to delay writing an actual article until you've replicated the significant results in a more focused study.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.