Discrepancy between ANOVA analyses

Question

My topic is faking emotional intelligence. I have a between-subjects design with three conditions. People were asked to complete the emotional intelligence survey in either of these conditions:

1. as a job applicant
2. being self-critical
3. no instructions (considered "honest condition").

An ANOVA did indeed show that people do fake: those in the job applicant stage had significantly higher EI scores than those in the honest condition; this is how I have defined faking.

I am most interested in seeing whether there are gender differences in faking EI. From a line graph, I can see that women fake more than men (women have much higher scores in condition 1 vs condition 3; men have a smaller gap between scores in condition 1 vs 3). From the following three analyses, it appears that women do fake more than men: 1) As mentioned, those in the job applicant condition have significantly higher scores than those in the honest condition 2) Filtering for only the job applicant condition, an ANOVA reveals that women score significantly higher on EI than men 3) Filtering for only the honest condition, an ANOVA shows that there are no significant differences in EI scores between men and women

BUT, the problem is I'm not sure if what I described above is essentially like conducting a two-way ANOVA with gender and condition as IVs and the EI scores as the DV. I had thought of this initially and ran this ANOVA but failed to get a significant gender X condition interaction (filtered just for conditions 1 and 3).

My question is: based on the two ways I approached my question about gender differences, which is correct? My three-step analysis which infers that women do fake more? Or the two-way ANOVA which shows no significant interaction result, suggesting no gender differences in faking?

Answer 1

What you're trying to do here is estimate the difference between women & men in the difference between the job-applicant & no-instructions conditions in emotional intelligence score. Differences of differences are interaction effects, & must be analysed as such—it's the whole point of setting up a controlled experiment as you have done.

Consider:

Filtering for only the job applicant condition, an ANOVA reveals that women score significantly higher on EI than men (p=0.049)

Filtering for only the honest condition, an ANOVA shows that there are no significant differences in EI scores between men and women (p=0.051)

The two effects could be almost the same size.

NB No significant interaction does not suggest no differences in faking. It only means your experiment was unable to tell you at a given significance level whether women or men fake more. Look at the confidence interval for the interaction effect to assess whether there may or may not be any important difference between women & men in this respect.