3
$\begingroup$

Let's say we have three intervention groups (condition A, condition B, and condition C/control) to be analyzed in ANOVA, and we are theoretically interested in the difference between each pair of all the possible comparisons (i.e., A and B, B and C, and A and C). Since the three comparisons are according to the hypotheses, I thought that it would be ok to describe this as planned comparisons. However, the planned comparisons do look like post-hoc tests (that test all possible comparisons for exploratory purposes). Is it still acceptable to consider it as planned comparisons without the necessity to adjust the p values? Or should I regard it as post-hoc tests and adjust the p values for multiple testing?

Summary of the suggestions:

  • I am starting to understand that what I described is not really how planned comparisons should be done. When I run planned contrasts, I may start from comparing intervention groups (A & B) vs. control group (C), followed by the comparison of the two intervention groups (A vs. B). Alternatively, I can focus on a specific pair, although the maximum number of contrasts should be k-1, as BruceET has suggested. (15/Sep/2021)
  • Thank you so much for your brilliant insights, Tanner! It is important to adjust significance level to avoid false findings when running multiple tests. Together with COOLSerdash's critical suggestions, I may not adjust the significance level when I have a specific hypothesis for each comparison (17/Sep/2021).
$\endgroup$
2
  • 1
    $\begingroup$ If you have a factor with $k$ levels, then that factor has $k-1$ degrees of freedom and you should be able to find $k-1$ orthogonal contrasts, all of which can be tested as a package of "pre-planned" orthogonal contrasts. [For exmp: If you have 3 levels ABC, then you could compare 'A vs B' and 'avg of A&B vs C'. Maybe google "orthogonal preplanned contrasts".] If you want to make more comparisons or non-orthogonal ones, the usual procedures of avoiding false discover in ad hoc testing apply. // A separate issue is how many of the $k-1$ orthogonal contrasts are of practical interest to you. $\endgroup$
    – BruceET
    Sep 14, 2021 at 2:17
  • 1
    $\begingroup$ Thank you so much for your advice! I am beginning to understand that I should do A & B vs. Control first. Then, I could move on to A vs. B. So it becomes two contrasts in total at most (3 factors -1 = 2 contrasts/degrees of freedom), out of which I can choose the contrasts of interest. $\endgroup$
    – Atsushi
    Sep 15, 2021 at 6:04

2 Answers 2

3
$\begingroup$

It's not only possible, it's explicitly recommended (see Ruxton & Beauchamp 2008).

First, if the omnibus-test of homogeneity across all groups is not of interest (e.g. the overall ANOVA $F$-test), consider not paying attention to it at all or not doing it in the first place. Second, if each planned comparison tests a different specific hypothesis, it is actually controversial if a formal control of the experimentwise type 1 error rate (EER) is required or not. Some text do not consider it necessary (Kirk 1995, Quinn & Keough 2002, Rothman 1990, Rubin 2021, Sokal & Rohlf 1995). Ruxton & Beauchamp (2008) recommend not controlling EER if the set of pre-planned contrasts is orthogonal. If all possible pairwise comparisons between groups are planned, the set of contrasts is not orthogonal and so, Ruxton & Beauchamp (2008) recommend controlling EER.

Rubin (2021) on the other hand argues that in the case of individual testing, no alpha adjustment should be done. He defines individual testing as tests, where each individual result must be significant in order to reject each associated individual null hypothesis. This seems to be the case here if you want to test each pairwise group difference individually. Personally, I find his arguments convincing and have subsequently changed my own opinion on the matter.

References

Kirk RE. 1995. Experimental design. Pacific Grove (CA): Brooks/Cole.

Quinn GP, Keough MJ. 2002. Experimental design and data analysis for biologists. Cambridge (UK): Cambridge University Press.

Rothman, K. J. (1990). No adjustments are needed for multiple comparisons. Epidemiology, 43-46. (link)

Rubin, M. (2021). When to adjust alpha during multiple testing: A consideration of disjunction, conjunction, and individual testing. Synthese, 1-32. (link)

Ruxton, G. D., & Beauchamp, G. (2008). Time for some a priori thinking about post hoc testing. Behavioral ecology, 19(3), 690-693. (link)

Sokal RR, Rohlf FJ. 1995. Biometry. 3rd ed. New York: WH Freeman.

$\endgroup$
3
  • $\begingroup$ Thank you so much for your detailed comments on this topic and sharing your references, COOLSerdash. I see this is a hotly debated topic and will need to learn about whether/when control of significance level becomes necessary. $\endgroup$
    – Atsushi
    Sep 16, 2021 at 6:45
  • $\begingroup$ @Atsushi If you're satisfied with an answer (mine or any other), consider upvoting it by clicking on the "arrow up" symbol on the left. If you're feeling that this answers your question fully, consider accepting it by clicking on the check mark. $\endgroup$ Sep 16, 2021 at 7:52
  • 1
    $\begingroup$ Ok, I did not know that! I have done so for the people who contributed to this topic. Thank you so much for you comments and suggestions! $\endgroup$
    – Atsushi
    Sep 17, 2021 at 3:08
2
$\begingroup$

The need for P-value adjustments is not due to "planning vs. post-hoc" but about alpha inflation. If I understand your analysis plan correctly, you would like to do pairwise tests comparing the relationship between each pair: AB, BC, AC.

At an alpha of 0.05 (i.e. p-value threshold of .05) you are excepting a 5% risk of there NOT actually being a statistically significant difference between the groups even though you said there was. So if you run three tests, you are actually taking a 5% risk three times, and your "true" risk level is no longer 5%, it's closer to 15%.

This has nothing to do with whether you planned these tests ahead of time or not-- it is simple a matter of re-testing the same data 3 times. So, yes, you should use p-value adjustments.

Note: the true test risk is not exactly 15%. It can be calculated with the binomial distribution, but the intuition seemed sufficient to answer this question.

$\endgroup$
2
  • 1
    $\begingroup$ I’d tweak this to remove the mention of “statistically significant” in the second paragraph. (Do you see why?) Then this would get a +1 from me. $\endgroup$
    – Dave
    Sep 14, 2021 at 2:16
  • 1
    $\begingroup$ Then, why do researchers recommend we consider significance adjustment when there are no preplanned hypotheses? onlinelibrary.wiley.com/doi/full/10.1111/opo.12131 $\endgroup$
    – Atsushi
    Sep 15, 2021 at 4:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.