Suppose I'm running a 3x3 repeated measures ANOVA with the main effects of "Condition" and "Species": Condition (hat x neutral x glasses) Species (dog, cat, human). The experiment involves human subjects who rate images ("How cute do you think a dog in a hat is" according to their cuteness.)
In this research, I want to see if there are main effects of "Condition" or "Species" on perceived cuteness. However, I don't want to compare a dog wearing a hat to a human wearing glasses, because it's like comparing apples and oranges (suppose according to this theory you can't compare a dog with a neutral condition with a human in a hat). Instead, I want to compare a dog wearing a hat to a cat or a human wearing a hat, to see which species or conditions are perceived as cutest.
However, many "reputable" names in the literature have done it this way (with 3x3 anova). However, this does not seem very right to me. To me, this seems like it ignores what I want to compare. For example, the interaction effect is significant even if it is not significant for what I want to compare in post-hocs (e.g. cat with glasses x dog with the hat is significant. But a cat with glasses x dog with glasses or cat with hat x dog with the hat is not significant). Sometimes the interaction effect is not significant but the post-hocs test shows significant (with Tukey correction).
It might be more appropriate to do a separate ANOVA for each "condition" (e.g. only for species wearing hats) to see which species or conditions are perceived as cutest. However, if I do it this way, I won't be able to see which condition/species is more important overall.
For example, when I do a 3x3 ANOVA, I see that the "condition * species" interaction effect is not significant in the model. And here is a post-hoc example from this 3x3 model:
Post-hoc from a 3x3 anova:
Cat with hat ----------- Dog With Hat--------p=.13
If I conduct another repeated measure anova for only hat condition :
Cat---------------------- Dog p=.00------
I found the above answer is very informative but I'm not exactly sure since 10+ literature still conducting a 3x3 anova. I don't know how can I justify my claim.
I know it's hard to interpret such questions without seeing data. I tried my best to describe my problem.