I am analyzing the data of a personality survey to see whether the order in which a text was presented to participants has an effect on their scores.
For that, I am performing a one-way independent measure ANOVA with three groups (text presented 1st, 2nd or 3rd), on each of the five personality trait scores I compute from the participants. For 4 traits (i.e. in 4 ANOVAs) no significant effect between the groups can be found (which is what I expected). For one trait, however, I get a significant difference in mean values. I performed a post-hoc pair-wise Tukey HSD test and determined that only one of the three possible pairings shows significant differences.
This is a somewhat weird result, as it suggests that only neuroticism is affected by the order of my text, and only when the text is presented 3rd. I am drawn to the following interpretation: an ANOVA with 3 groups, which shows no significant results, is effectively conducting three pair-wise comparisons. I performed 5 ANOVAs, which would be comparable to 15 pair-wise tests. With an alpha level of 0.05 I would expect to get at least on false-positive in 20 tests, so finding one in 15 would not be unlikely. My conclusion: the effect I found was a random sampling effect, not an effect of group order.
However, I am not sure about my first assumption: "an ANOVA with 3 groups, which shows no significant results, is effectively conducting three pair-wise tests". Is this a warranted interpretation?