2
$\begingroup$

I am currently analyzing data for an age-related neuroscience study with a 2x2 factorial design (treatment (2 levels) x age (2 levels)) yielding 4 total groups: Young untreated, Young treated, Aged untreated, Aged treated. The hypothesis is that the two aged groups would be different, while young animals show no difference. My first thought is to perform a factorial ANOVA (test for main effect of genotype and age, as well as test for an interaction effect). However, if there is no interaction effect of the two variables, I'd still like to test that original hypothesis (i.e., compare just the aged groups and just the young groups). I've recently learned that some researchers employ planned comparisons (I've also seen it called planned contrasts) to examine differences between specific groups that were selected a priori instead of doing an ANOVA and running post-hoc tests that examine all possible comparisons.

This makes sense to me, but I've heard that scientific journals "don't like" when researchers use planned comparison, so getting published with planned comparisons would be difficult if not near impossible. On the other hand, through very cursory internet searching, I found 2 papers that seem to advocate for the use of planned comparisons since every study should have a hypothesis that predicts specific effects anyway (The papers can be found here and here).

Given all this, my question is as follows: Is it generally acceptable to do planned comparisons if there's a rational justification for it, or is this approach frowned upon by journals and peer reviewers?

Improve this question
$\endgroup$
1
  • $\begingroup$ This was posted on Academia.SE earlier, and I said that it should be here, as it ends up being a statistical question. // My take on this is that prespecified hypotheses are preferable, as post-hoc hypotheses because "oh, what's going on there?" means that there are issues with adjusting for multiple comparisons. (That's true for prespecified hypotheses, too, but you go into the work knowing how many hypotheses there are.) $\endgroup$
    – Dave
    Commented Mar 23, 2021 at 15:14

1 Answer 1

Reset to default
1
$\begingroup$

I've heard that scientific journals "don't like" when researchers use planned comparison, so getting published with planned comparisons would be difficult if not near impossible.

I haven't heard that myself. There is much to be said statistically for starting with clear, pre-planned hypotheses to test rather than just blindly jumping into ANOVA with all paired comparisons evaluated post-hoc. I suspect that journals who "don't like" pre-planned comparisons have pretty poor statistical review. (Or perhaps they fear that the "pre-planned" hypotheses were really developed post-study.)

With this 2 x 2 design, however, there won't be much of a difference. Under the principle of marginality, if there is an interaction there aren't well-defined unique "main effects" for either age or treatment. If you find a significant interaction, then the planned within-age-group comparisons of the treatment effect make sense. If, however, the interaction term is not significant, then there is no evidence that the effect of treatment differs with age so you should just stop there.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for this answer and the link. I am still a bit confused, however, because I'm interpreting your second paragraph to suggest that pre-planned comparisons in this instance are essentially no different than post-hoc tests (i.e., without a main effect of interaction you can't do the pre-planned within-age comparison). Is that correct? Also, what if one factor shows a main effect while there is no main effect of of the other nor interaction? Would a pre-planned comparison be suitable to parse that out in the context of the original hypothesis? $\endgroup$
    – user120818
    Commented Mar 23, 2021 at 16:31
  • $\begingroup$ @user120818 if the overall ANOVA isn't significant you should stop, as there is no evidence that anything you've examined is associated with outcome. If there is a significant ANOVA but not a significant interaction, you could report the marginal means for each of age and treatment as their "effects"; the R emmeans package makes that easy. If there is a significant interaction too, then any pre-planned comparisons would be OK. Beyond that, "won't be much of a difference" meant there are only 6 pairwise differences in a 2 X 2 design, anyway. $\endgroup$
    – EdM
    Commented Mar 23, 2021 at 17:06
  • $\begingroup$ Thank you for the clarification. Just to make sure I have a handle on pre-planned comparisons vs. post-hoc analyses, I have a couple more questions. Essentially, what separates post-hoc from pre-planned comparisons if both require a significant interaction via ANOVA to assess the groups of interest? Is it simply that pre-planned comparisons don't necessarily need to correct for multiple comparisons? I'm mainly getting tripped up on what the purpose of a planned comparison is if the scenarios in which you would run them are the same as a post-hoc. Is it to increase the statistical power? $\endgroup$
    – user120818
    Commented Mar 23, 2021 at 17:24
  • $\begingroup$ I think I may have found the answer to in another question posted to this forum: stats.stackexchange.com/questions/342638/…, but I'm still interested to get your thoughts, @EdM $\endgroup$
    – user120818
    Commented Mar 23, 2021 at 17:52
  • $\begingroup$ @user120818 yes, that's a very helpful link, glad you found it. The main issue is how many multiple comparisons you need to correct for. In a 2 X 2 design, there are 6 potential pairwise comparisons. If you pre-plan fewer comparisons/contrasts, there is less stringent multiple-comparison correction needed and thus potentially higher power. $\endgroup$
    – EdM
    Commented Mar 23, 2021 at 18:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.