22
$\begingroup$

I'm reviewing a paper that has performed >15 separate 2x2 Chi Square tests. I've suggested that they need to correct for multiple comparisons, but they have replied saying that all the comparisons were planned, and therefore this is not necessary.

I feel like this must not be correct but can't find any resources that explicitly state whether this is the case.

Is anyone able to help with this?

Update:

Thanks for all of your very helpful responses. In response to @gung's request for some more information on the study and the analyses, they are comparing count data for two types of participants (students, non-students) in two conditions, across three time periods. The multiple 2x2 Chi Square tests are comparing each time period, in each condition, for each type of participant (if that makes sense; e.g. students, condition 1, time period 1 vs time period 2), so all analyses are testing the same hypothesis.

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Many people that carry out multiple comparisons plan to do all of them a priori. They do it because they want to control the overall type I error rate. In some situations it can be reasonable to not correct for multiple comparisons, but it's not just a matter of planning to do all of them from the start. $\endgroup$
    – Glen_b
    Commented Jan 10, 2017 at 11:12
  • 3
    $\begingroup$ Can you say a little more about the study, their data, & there analyses? Do the >15 amount to all possible comparisons, or only a small %? How much data do they have? How plausible is it that the hypotheses were all a-priori? Are they all significant? Are the chi-squared tests independent of one another? Also consider some of the questions raised in @peuhp's answer. $\endgroup$
    – gung - Reinstate Monica
    Commented Jan 10, 2017 at 19:13
  • 5
    $\begingroup$ Because "they" are likely interested in finding significant results, their response is self-serving. Therefore the burden is on them to demonstrate why their approach is legitimate, rather than on you to show it is illegitimate. Any attempt to show that multiple comparisons corrrections can be neglected will fail as soon as it considers the paper-wide false positive rate, and therefore "they" must either (disingenuously) avoid all consideration of that issue or else provide a good argument concerning why it is of no concern for their intended audience. $\endgroup$
    – whuber ♦
    Commented Jan 10, 2017 at 19:21
  • 2
    $\begingroup$ I would be sorely tempted to respond with a link to this XKCD strip (which, as you might note, involves a fully planned series of multiple tests...). $\endgroup$
    – Ilmari Karonen
    Commented Jan 11, 2017 at 9:37

4 Answers 4

Reset to default
25
$\begingroup$

This is IMHO a complex issue and I would like to make three comments about this situation.

First and generally, I would more focus on whether you face a confirmatory study with a set of well-shaped hypotheses defined in a argumentative context or an explanatory study in which many likely indicators are observed than whether they are planned or not (because you can simply plan to make all possible comparisons).

Second, I would also focus on how the resulting p-values are then discussed. Are they individually used to serve a set of definitive conclusions, or are they jointly discussed as evidence and lack of evidence?

Finally, I would discuss the possibility that the >15 hypothesis resulting from the >15 separate chi-squared tests are in fact the expression of a single few hypotheses (maybe a single one) that may be summarized.

More generally, regardless of whether hypothesis are prespecified or not, correcting for multiple comparisons or not is a matter of what you include in the type I error. By not correcting for MC, you only keep a per comparison type I error rate control. So in case of numerous comparisons, you have a high family-wise type I error rate and thus are more false discovery prone.

Improve this answer
$\endgroup$
5
  • 9
    $\begingroup$ (+1) It might be worth spelling out that the experiment-wise error rate is not controlled by the fifteen individual comparisons' being planned; on the other hand, possible comparisons beyond the fifteen not envisaged in the protocol need not be taken into account in multiple-comparisons correction. $\endgroup$
    – Scortchi ♦
    Commented Jan 10, 2017 at 10:43
  • $\begingroup$ @Scortchi Thanks for your input but I do not understand what do you mean by "experiment-wise error rate is not controlled by the fifteen individual comparisons' being planned" ? $\endgroup$
    – beuhbbb
    Commented Jan 10, 2017 at 12:55
  • 1
    $\begingroup$ Just the basic point that if you want to control the probability under the null of making one or more Type I errors across all those tests you do need to use a multiple comparisons procedure. I only mention it because I've come across confusion on the matter before. $\endgroup$
    – Scortchi ♦
    Commented Jan 10, 2017 at 13:09
  • 2
    $\begingroup$ Note that this exact same issue came up in a very recent thread: Post Hoc application of Multiple Comparisons. $\endgroup$
    – Michael R. Chernick
    Commented Jan 10, 2017 at 13:20
  • 1
    $\begingroup$ @Scortchi. Ok thanks for this clarification and input, this should indeed be clearly specified in my answer. Will add this. $\endgroup$
    – beuhbbb
    Commented Jan 10, 2017 at 13:22
5
$\begingroup$

If you substitute the word 'premeditated' for 'planned', this may help dispel the argument offered by the authors. Consider two different statistical analyses of the same data:

  1. A 'premeditated crime' in which every possible hypothesis test is laid out combinatorially in advance by a 'statistical criminal mastermind', the plan being to try each one systematically, and pick the test with the smallest p-value as the 'key finding' to promote in the Results, Discussion and Conclusion sections of the paper, and indeed the Title as well.
  2. A 'crime of passion' in which the initial intention was merely to confront the data with one hypothesis, but "well...one thing leads to another" and multiple ad hoc hypothesis tests "just happen" in the heat of scientific passion to learn "something ... anything!" from the data.

Either way, it's 'murder' — the question is whether it's in the First Degree or Second Degree. Clearly, the first is morally more problematic. It sounds to me as if the authors here are attempting to claim something to the effect that it wasn't murder because it was premeditated.

Improve this answer
$\endgroup$
1
  • 4
    $\begingroup$ But doing multiple comparisons is not a crime, premeditated or not. P-hunting is. $\endgroup$
    – Cliff AB
    Commented Jan 10, 2017 at 23:45
5
$\begingroup$

Given your update on the design I would suggest that they do some form of log-linear model to use all of the data at once. Doing the piece-meal analyses they have done seems (a) inefficient (b) unscientific as it tests 15 hypotheses where surely there are fewer real hypotheses.

I am not a fan of correcting for multiplicity as a conditioned reflex but in this case if they reject a deeper analytic approach then I would suggest they correct.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ If all tests test the same hypothesis, then one can also use some meta-analytic tool to "combine" the 15 tests' results into one. You are an expert on meta-analysis so perhaps you could suggest something more specific. As the simplest thing, I've seen people computing the p-value for getting $k$ significant outcomes out of $15$ tests; but this assumes independence which in OP's case is obviously not true. $\endgroup$
    – amoeba
    Commented Jan 12, 2017 at 13:13
  • 1
    $\begingroup$ @amoeba I think that would be a last resort as I cannot help feeling there must be a better way of analysing this than 15+ $\chi^2$ tests. $\endgroup$
    – mdewey
    Commented Jan 12, 2017 at 13:53
1
$\begingroup$

This paper directly addresses your question: http://jrp.icaap.org/index.php/jrp/article/view/514/417

(Frane, A.V., "Planned Hypothesis Tests Are Not Necessarily Exempt From Multiplicity Adjustment", Journal of Research Practice, 2015)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.