Fishy significance test: Am I doing something wrong with ANOVAs? Just when I thought I'd had a grip on how to do an analysis of variance this particular data set had me startled: it's a collection of response times (in ms) to a linguistic input. To be precise, it's part of a reading time experiment, and I'm trying to see if there's a significant effect of two factors.
My experiment had a 2x2 factorial design, with 2 factors binary Conflicting and ContextPresent. The only value I'm interested in for the purpose of this question is one particular random variable, a response time. No transformations have been done on it, except removal of outliers via 3-sigma rule (I used mean + 3 * standard deviation to determine outliers.)
So I run my anova in R:
> anova(lm(TextDisplay9.RT ~ Conflicting * ContextPresent, data=items.cropped))
Analysis of Variance Table

Response: TextDisplay9.RT
                            Df   Sum Sq Mean Sq F value  Pr(>F)
Conflicting                  1   111185  111185  7.0591 0.00808 **
ContextPresent               1    73591   73591  4.6723 0.03102 *
Conflicting:ContextPresent   1      352     352  0.0223 0.88128
Residuals                  651 10253667   15751

Jolly ho, I get pretty good results for a main effect on both my factors, and no interaction. That's fine. But here's the corresponding box-and-whiskers graph:
> ggplot(items.cropped,aes(Conflicting,TextDisplay9.RT)) + geom_boxplot() +
  facet_grid(.~ ContextPresent)


So this plot actually makes it seem like there shouldn't be a main effect of either variable! They're all too similar! Yes, the scale is rather squished, because of the outliers, but the means are really really close!
> with(items.cropped,mean(items.cropped[Conflicting=="semantic conflict" & ContextPresent == "mentioned in context",]$TextDisplay9.RT,na.rm=T))
[1] 431.8659
> with(items.cropped,mean(items.cropped[Conflicting=="semantic conflict" & ContextPresent == "not mentioned in context",]$TextDisplay9.RT,na.rm=T))
[1] 454.5305
> with(items.cropped,mean(items.cropped[Conflicting=="no semantic conflict" & ContextPresent == "mentioned in context",]$TextDisplay9.RT,na.rm=T))
[1] 407.485
> with(items.cropped,mean(items.cropped[Conflicting=="no semantic conflict" & ContextPresent == "not mentioned in context",]$TextDisplay9.RT,na.rm=T))
[1] 427.2188

Does it sound possible that there could be a main effect? Or did I somehow misuse ANOVAs? I could provide the data if needed!
Thanks very much for any suggestions.
 A: The box-and-whisker plots look very skewed, and that is after removing the outliers.
My suggestion: Transform each value to its reciprocal. Now instead of assessing how long something takes, you will be assessing how fast it is. Now look at the distribution of values (before removing any outliers). My guess is that the data will be far more symmetrical, and you'll have many fewer outliers. Also my guess is that the data will be closer to Gaussian, which means an ANOVA on the transformed values will be more valid than the original ANOVA.
Also note that while the P value was small in your ANOVA for the "conflicting" factor, the effect size is tiny. The R squared is computed from the ratio of SS, so is only 111185/10253667 = 0.0108.
A: The lines in the boxplots are medians not means.  When you have outliers the mean and medians can be very different.  Did you include the outliers when fitting the model?  If you excldued them what was your justification for removing them?  Also if the outliers are not used in the model you get a misleading picture including them in the boxplot. The distributions look skewed, particularly the one on the far right and far left.  Observations should not be arbitrarily removed for exceeding the 3 sigma limit. When the data is very nonnormal the F tests in the analysis of variance are not valid and you should consider nonparametric alternatives such as the Kruskal-Wallis test.  Also keep in mind that even for a normal distribution approxiamtely 3 out of 1000 observations will fall outside the 3 sigma limit.  With a sample of over 650 observations it would not be surprising to have one or two outside the 3 sigma limit.
A: Give a better justification for outlier removal than 3 SD.  Items happen at 3 SD by chance, at a pretty high probability, when you have this many samples.  There are also many other issues (see Miller (1991) for an example).  There may be improbable values based on theory (e.g. impossibly fast, or ridiculously slow).  Remove outliers because of that.  Don't use an arbitrary (which is what this is) statistical procedure over your own judgment.  What if an RT of 80 ms is not 3SD away?  Do you keep it?  It's not physically possible to be actually based on perception of a stimulus for button presses or vocal responses.  With a choice task values like the 190 ms in your plot would not be believable (and that looks like it's post outlier removal).  You mention several seconds must be outliers, then remove them for that reason. They must reflect processes other than a response to the stimulus.  If you have an accuracy measure you can use those to guide outlier removal.  Perhaps accuracy is very low under a certain RT or drops off past a different RT.
You are not allowed to use all of the degrees of freedom in all of your measurements in single level ANOVA because this is a repeated measures design.  You must aggregate your numbers so that each subject produces 4 numbers, 1 value in each condition.
items.agg <- aggregate(TextDisplay9.RT ~ Conflicting + ContextPresent + Subject, itemsitems.cropped, mean)

If you examine these aggregate data you may find your skew problem is solved through the central limit theorem (although the n is a bit low for that).  It would almost definitely be solved for the reciprocal of the RT in seconds (rate).  RT is an arbitrary representation of performance.  It is the time it took to complete the task.  The rate would be how many tasks can be completed per second.  They are both easily interpretable numbers but the latter has much better statistical properties. (do not forget that the meaning of rate is opposite of RT in that higher numbers = better performance)
And then you have to stratify your results in the repeated measures ANOVA
m <- aov(TextDisplay9.RT ~ Conflicting * ContextPresent + Error(Subject / (Conflicting * ContextPresent)), data = items.agg)
summary(m)

You'll have much less power in your study now.
You really should look at Baayen's (2008) book on studying linguistic RT data.  It's very specific to your field and avoids much of the messy statistical theory while being very practically helpful.
Miller, J. (1991). Reaction time analysis with outlier exclusion: Bias varies with sample size. The Quarterly Journal of Experimental Psychology, 43A(4):907–912.
