0
$\begingroup$

I am analyzing the data of a personality survey to see whether the order in which a text was presented to participants has an effect on their scores.

For that, I am performing a one-way independent measure ANOVA with three groups (text presented 1st, 2nd or 3rd), on each of the five personality trait scores I compute from the participants. For 4 traits (i.e. in 4 ANOVAs) no significant effect between the groups can be found (which is what I expected). For one trait, however, I get a significant difference in mean values. I performed a post-hoc pair-wise Tukey HSD test and determined that only one of the three possible pairings shows significant differences.

This is a somewhat weird result, as it suggests that only neuroticism is affected by the order of my text, and only when the text is presented 3rd. I am drawn to the following interpretation: an ANOVA with 3 groups, which shows no significant results, is effectively conducting three pair-wise comparisons. I performed 5 ANOVAs, which would be comparable to 15 pair-wise tests. With an alpha level of 0.05 I would expect to get at least on false-positive in 20 tests, so finding one in 15 would not be unlikely. My conclusion: the effect I found was a random sampling effect, not an effect of group order.

However, I am not sure about my first assumption: "an ANOVA with 3 groups, which shows no significant results, is effectively conducting three pair-wise tests". Is this a warranted interpretation?

$\endgroup$
3
$\begingroup$

If an ANOVA test showed no statistically significant results, you should ignore any post hoc pairwise tests.

On the other hand, even if an ANOVA test shows statistically significant result, post hoc significance is not guaranteed.

Edit: ANOVA is an omnibus test. Short version: ANOVA tests the average score of each group with the average score of all groups, taking into account the size of each group and the variance of each group. It's just one Sum (Σ). So, due to "blindness/chance/randomness" you would have 5% chance for false positive for each ANOVA.

$\endgroup$
  • $\begingroup$ I perform no post-hoc tests on the traits where ANOVA shows no effect. I do find significance in one post-hoc pairwise test on the three means where ANOVA shows an effect. My interest is rather in how to interpret what an ANOVA does in general: Does it equal to comparing all means to each other pairwise, or not? If this were the case, I would have conducted 15 tests on my data, if not, only 5. This is relevant because seeing one false positive in 15 tests is more probable than seeing one in 5 tests. $\endgroup$ – Zakum Feb 15 '18 at 11:08
  • 1
    $\begingroup$ ANOVA is an omnibus test. Short version: ANOVA tests the average score of each group with the average score of all groups, taking into account the size of each group and the variance of each group. It's just one Sum (Σ). So, due to "blindness/chance/randomness" you would have 5% chance for false positive for each ANOVA. $\endgroup$ – AchiPapakon Feb 15 '18 at 22:19
  • $\begingroup$ Accepting this answer for the reference to omnibus test in your last comment. Thanks, and feel free to put this as an edit to the actual answer for better visibility. :) $\endgroup$ – Zakum Feb 19 '18 at 13:15
1
$\begingroup$

I don't think that is a valid interpretation, Zakum. The post-tests compare pair-wise between groups. The original ANOVA will compare the three groups, in a one way ANOVA, on whether you can say that the means are drawn from different populations. That is my interpretation. Adding a second factor, the personality traits, will require additional ANOVA. In a one-way ANOVA, the F value is computed as (Sum of Squares due to Treatments) / (Sum of Squares due to Error), so I don't think you can make the assumption about the pair-wise. I think that is what the post-tests are for. I am by no means a stats expert though.

EDIT: Building on the comment, if I understand what you are asking, there were three ANOVAs performed separately on 1&2; 2&3; 1&3; then I think it entirely possible that you could see one set as statistically different while the other sets have p>.05 and so to your original question, the ANOVA with 3 groups is comparing the 3 means, and the post-tests are 3 pair-wise comparisons. As far as I understand it, the F value in the original ANOVA with 3 groups is different than the t value in the post-tests, meaning I don't think the F is a compilation of 3 pair-wise t values directly, if that makes sense. If I understand your experimental design, then the text presented 3rd affects the neuroticism trait statistically different than text presented 1st. Text presented 2nd may be moving the 'score' of the neuroticism trait, but it is not statistically different until it reaches the 3rd position? So then I want to ask "what if there was a fourth in order option for the text, would it be more significant?"

$\endgroup$
  • 1
    $\begingroup$ Thanks for your insights. Your words about samples being drawn from the same population makes me wonder: my post-hoc pairwise comparison showed the following results: mean1 and mean2 are from the same population (no significant differences), mean2 and mean3 are from the same population (no significant differences), BUT: mean1 and mean3 are from different populations (P=0.028). Being drawn from the same population should be a transitive property (m1, m2 from same pop and m2,m3 from same pop -> m1,m3 from same pop). Any ideas how to interpret the finding that this doesn't hold? $\endgroup$ – Zakum Feb 15 '18 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.