4
$\begingroup$

I have been struggling with the idea of multiple comparisons for while. Correcting for multiple comparisons is easy to understand and to correct for when conducting for example an ANOVA between several variables during a posthoc test. However, how do you correct between tests, if you have several separate hypotheses?

For example, I have a dataset with two main a priori hypotheses when setting up the study. Later, when thinking about any interesting situations, I thought about splitting the data, or combining groups etc. and looking into more questions. They all would answer separate questions, but I end up with about 20 separate tests (t-tests and ANOVAs). Clearly, this is exploratory, since I did not specify these tests beforehand, since I was mainly interested in the two a priori hypotheses. Should I still correct for multiple comparisons and how do you generally do that between say 4 t-tests and 6 ANOVAs?

Is doing this many tests fishing and general bad practice? Should I ignore it completely, or could I simply call it exploratory?

NOTE: The data I use has 4 groups of about 5 subjects per group. I have, depending on combining or splitting the data, about 4 dependent variables of interest. I can group the 4 groups in several meaningful ways together, same goes for the dependent variables. Therefore I have a huge amount of options on how to look into the data.

$\endgroup$
2
$\begingroup$

However, how do you correct between tests, if you have several separate hypotheses?

There is a lot of debate going on right now about this. There is almost a consensus now that you shouldn't just go fishing around in your data for significant p-values, as this inflates Type I error (obviously).

However, it is also true that effect sizes are not a product of any previous analyses you have done—they are only products of the data. So why would a test you do 20th be less believable than the one you happened to do 1st because of your a priori thinking?

Obviously, there is a tension here.

Should I still correct for multiple comparisons and how do you generally do that between say 4 t-tests and 6 ANOVAs?

There is no straightforward or agreed upon way of doing this. There are any number of corrections for post hoc comparisons (Bonferroni, HSD, Holm, whatever). A professor of mine had really great advice in saying that: "If there are many ways of doing something, it is because there is no best way of doing it." Or, at least, there is no widely agreed-upon way of doing it.

You could simply divide .05 by 10 (the number of tests you did total, similar to a Bonferroni correction) to be very conservative. However, this might give you a Type II error simply because you were wrong a priori, which we don't want either!

Is doing this many tests fishing and general bad practice?

Collecting data, not finding what you wanted a priori, and then digging around, doing dozens of tests, and then finding something significant is not bad practice. HOWEVER, it becomes bad practice when you communicate the finding to an audience like it was the very first test you did, like you had predicted it the whole time, and hiding any other tests you did beforehand. This becomes a disastrous practice if you are hiding a test you did beforehand that might contradict the significant finding you found.

Should I ignore it completely

No, but be more skeptical than you generally would (I hope you are always skeptical of what one study finds).

could I simply call it exploratory?

Yes. It is OK to include this analysis in a paper. However, be very upfront about how many tests you did beforehand, that you hadn't initially predicted it, and report any analyses that you did in the data that could contradict whatever you found.

Also, do not take one study too seriously. I forgot who said it first—I believe it has been attributed to many people—but the saying goes that, "An ounce of replication is worth a ton of inferential statistics." Instead of getting so in a fuss about how to correct a p-value, why not take the interesting finding you had and replicate it with a well-powered sample?

If I were reviewing a paper and someone says, "We dug around in the data and found X. Here is the study that shows X," I would likely reject.

However, if someone said, "We dug around in the data and found X. Here is a study that shows X. Now here are Studies 2, 3, and 4 that directly and conceptually replicate X." I would say, awesome! Who cares that you didn't predict it at first? You have now done a number of tests replicating it, showing less doubt that it was Type I error.

Overall: Dig around all you want. Get to know your data. But don't find p < .05, go run off and tell people only about that, pretending like you had the idea the whole time. If you find something interesting, think about it, put it in your back pocket, and try to replicate it if you think it is really worth something.

This is a hot-button issue, and I'm sure some may not agree with me. But I think what I've proposed is a reasonable way to approach exploratory analyses when correcting p-values isn't within a test (i.e., I just do HSD on post-hoc comparisons for an ANOVA), but between a number of different, conceptually different statistical tests.

$\endgroup$
  • $\begingroup$ Thanks a lot for the elaborate answer! It sounds all very reasonable and I will follow your advice! :) I'm actually now also a bit curious on what arguments those that don't agree with this come up with ;) $\endgroup$ – Inkling Jun 14 '17 at 20:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.