- Struggling with factorial three-way anova, in particular post-hoc tests and multiplicity control.
- Four continuous dependent variables.
- Three categorical between-subject independent variables:
- A (2 levels)
- B (2 levels)
- C (4 levels)
- Unbalanced groups: Median group size is 17, IQR is from 11 to 54, range is from 2 to 126.
- Study design: Exploratory observational.
- Research question: "Are there differences in any of the DVs across the levels of A overall, or within certain subgroups of B and C?"
- Analysis plan: Four 2x2x4 type II full factorial three-way anovas (one for each DV).
- Post-hoc plan: Pairwise comparison of A across all levels of the highest order significant interaction involving A.
- I'm using R for this analysis (trying to use emmeans for post-hoc tests).
My question: Is my plan any good and how exactly do I go about the post-hoc testing?
Research design and data
I want to examine the effects of three between-subject factors on four different independent variables. Two of the factors have 2 levels, let's call those factors A and B, and the third factor has 4 levels, let's call it C. I am mostly interested in the effect of A on each of the DVs, but also curious whether that effect might differ across levels of B and C. I don't want to test any specific hypotheses. The study design is observational, which is why the groups are unbalanced (mean group size is 35, median is 17, IQR is from 11 to 54, range is from 2 to 126). My research design is exploratory and I guess my research question could be formulated as "Are there differences in any of the DVs across the levels of A overall, or within certain subgroups of B and C?".
I am planning to run four 2x2x4 full factorial anovas (one for each DV). Since A is my main variable of interest, I'd like to follow up only on significant interactions involving A. More precisely, I'd like to conduct pairwise comparisons of A across all levels of the highest order significant interaction involving A. I'm using R Studio for this analysis and attempting to use the emmeans package for post-hoc tests.
Is the idea to run four Anovas appropriate given my research question, data structure and sample size?
Is a Type II Anova the right choice?
How exactly do I go about the post-hoc comparisons?
- 3.1 If I follow up on a significant three-way interaction with pairwise comparisons of A across all levels of BxC, how do I correct the family-wise error rate? I've been using emmeans and noticed that when I conduct pairwise comparisons, the p-value is only adjusted within each of the levels that I compare across. This means that for my analysis, since I would only be conducting one comparison (because A only has two levels) within each of the 2, 4 or 8 different levels (depending on which interaction of A with B and C are significant), no multiplicity corrections would be performed.
- 3.2 In emmeans, I also noticed that the degrees of freedom for each of the post-hoc comparisons is the same as the DF for the Anova. I understand this is by design because the comparisons are based on the overall model. However, I also don't really understand it. What would be more appropriate for my analysis? Using the overall model DF or calculating DF for each comparison?
- 3.3 Given a significant three-way interaction, would it be better to first examine the effects of AxB over each of the 4 levels of C, instead of jumping straight into the simple main effects of A across each of the levels of BxC?
Does any of this even make sense?
Sorry for the lengthy post. I appreciate any input. Thanks a lot!