0
$\begingroup$

Suppose I'm running a 3x3 repeated measures ANOVA with the main effects of "Condition" and "Species": Condition (hat x neutral x glasses) Species (dog, cat, human). The experiment involves human subjects who rate images ("How cute do you think a dog in a hat is" according to their cuteness.)

In this research, I want to see if there are main effects of "Condition" or "Species" on perceived cuteness. However, I don't want to compare a dog wearing a hat to a human wearing glasses, because it's like comparing apples and oranges (suppose according to this theory you can't compare a dog with a neutral condition with a human in a hat). Instead, I want to compare a dog wearing a hat to a cat or a human wearing a hat, to see which species or conditions are perceived as cutest.

However, many "reputable" names in the literature have done it this way (with 3x3 anova). However, this does not seem very right to me. To me, this seems like it ignores what I want to compare. For example, the interaction effect is significant even if it is not significant for what I want to compare in post-hocs (e.g. cat with glasses x dog with the hat is significant. But a cat with glasses x dog with glasses or cat with hat x dog with the hat is not significant). Sometimes the interaction effect is not significant but the post-hocs test shows significant (with Tukey correction).

It might be more appropriate to do a separate ANOVA for each "condition" (e.g. only for species wearing hats) to see which species or conditions are perceived as cutest. However, if I do it this way, I won't be able to see which condition/species is more important overall.

For example, when I do a 3x3 ANOVA, I see that the "condition * species" interaction effect is not significant in the model. And here is a post-hoc example from this 3x3 model:

Post-hoc from a 3x3 anova:

Cat with hat ----------- Dog With Hat--------p=.13

If I conduct another repeated measure anova for only hat condition :

Cat---------------------- Dog p=.00------

https://stats.stackexchange.com/a/593655/342487

I found the above answer is very informative but I'm not exactly sure since 10+ literature still conducting a 3x3 anova. I don't know how can I justify my claim.

I know it's hard to interpret such questions without seeing data. I tried my best to describe my problem.

Improve this question
$\endgroup$
7
  • 2
    $\begingroup$ You don't need to fit separate models for each "condition". It's better to fit the model you think is appropriate to all the data. Then look only at the comparisons of interest. Here is a recent CV thread about this same (or at least, very similar) point: Does the P value adjustment for Tukey method in emmeans differ between "between group" and "within group". $\endgroup$
    – dipetkov
    Commented Dec 10, 2022 at 16:17
  • $\begingroup$ @dipetkov Thank you for your insights. I still don't understand why the p-value changes if I conduct separate ANOVAs. Do you have an idea why the p-value with the Tukey adjustment is different in these different ANOVAs? $\endgroup$
    – AltunE
    Commented Dec 13, 2022 at 17:04
  • 1
    $\begingroup$ Why do you want to do two separate ANOVAs? I'm not sure why you have an expectation that if you perform two different analyses, you'll get the same outcome. Instead, perform the appropriate analysis: one model fitted on all the data. And then look only at the comparisons you care about. Do not remove terms from the model (eg. interactions) just because they are not significant. $\endgroup$
    – dipetkov
    Commented Dec 13, 2022 at 17:11
  • $\begingroup$ @dipetkov Thank you very much again. I already conducted my analysis in that way (one ANOVA and looked only at the comparisons that I wanted). But if I conduct different ANOVAs for each condition, I'm getting different p-values in post-hoc analysis. Is that impossible? did i do something wrong? $\endgroup$
    – AltunE
    Commented Dec 13, 2022 at 17:17
  • 1
    $\begingroup$ More seriously, there are dangers in running multiple analyses of the same dataset: it becomes more tempting for example to pick an analysis that gives a result which is easier to interpret or more consistent with scientific hypothesis (think of this as confirmation bias). That's one reason sometimes the analysis is pre-registered: you write down how you'll analyze the data before you run the experiment and collect the data. $\endgroup$
    – dipetkov
    Commented Dec 13, 2022 at 17:26

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.