# Fishy significance test: Am I doing something wrong with ANOVAs?

Just when I thought I'd had a grip on how to do an analysis of variance this particular data set had me startled: it's a collection of response times (in ms) to a linguistic input. To be precise, it's part of a reading time experiment, and I'm trying to see if there's a significant effect of two factors.

My experiment had a 2x2 factorial design, with 2 factors binary Conflicting and ContextPresent. The only value I'm interested in for the purpose of this question is one particular random variable, a response time. No transformations have been done on it, except removal of outliers via 3-sigma rule (I used mean + 3 * standard deviation to determine outliers.)

So I run my anova in R:

> anova(lm(TextDisplay9.RT ~ Conflicting * ContextPresent, data=items.cropped))
Analysis of Variance Table

Response: TextDisplay9.RT
Df   Sum Sq Mean Sq F value  Pr(>F)
Conflicting                  1   111185  111185  7.0591 0.00808 **
ContextPresent               1    73591   73591  4.6723 0.03102 *
Conflicting:ContextPresent   1      352     352  0.0223 0.88128
Residuals                  651 10253667   15751


Jolly ho, I get pretty good results for a main effect on both my factors, and no interaction. That's fine. But here's the corresponding box-and-whiskers graph:

> ggplot(items.cropped,aes(Conflicting,TextDisplay9.RT)) + geom_boxplot() +
facet_grid(.~ ContextPresent)


So this plot actually makes it seem like there shouldn't be a main effect of either variable! They're all too similar! Yes, the scale is rather squished, because of the outliers, but the means are really really close!

> with(items.cropped,mean(items.cropped[Conflicting=="semantic conflict" & ContextPresent == "mentioned in context",]$TextDisplay9.RT,na.rm=T)) [1] 431.8659 > with(items.cropped,mean(items.cropped[Conflicting=="semantic conflict" & ContextPresent == "not mentioned in context",]$TextDisplay9.RT,na.rm=T))
[1] 454.5305
> with(items.cropped,mean(items.cropped[Conflicting=="no semantic conflict" & ContextPresent == "mentioned in context",]$TextDisplay9.RT,na.rm=T)) [1] 407.485 > with(items.cropped,mean(items.cropped[Conflicting=="no semantic conflict" & ContextPresent == "not mentioned in context",]$TextDisplay9.RT,na.rm=T))
[1] 427.2188


Does it sound possible that there could be a main effect? Or did I somehow misuse ANOVAs? I could provide the data if needed!

Thanks very much for any suggestions.

• With a large enough sample size anything can be significant... – Dason Jul 11 '12 at 1:38
• Response times to linguistic input are often done with repeated measures. How many subjects in this experiment are there? How many trials / subject? – John Jul 11 '12 at 3:16
• @Dason, I did aggregate the data, and with the aggregated data I get F(1,92) = 6.67, p < 0.115, so it's still "signifiant." – Aleksandar Dimitrov Jul 11 '12 at 8:54
• @John, I had 7 subjects per condition/item pairing in this experiment. I read that in this case one might want to do anova(lm(val ~ f1 * f2 * Subject)), but that one gives me even better scores! – Aleksandar Dimitrov Jul 11 '12 at 8:55
• Given that you just have categorical predictors you have to use repeated measures ANOVA or multi-level modelling. – John Jul 11 '12 at 12:04

Give a better justification for outlier removal than 3 SD. Items happen at 3 SD by chance, at a pretty high probability, when you have this many samples. There are also many other issues (see Miller (1991) for an example). There may be improbable values based on theory (e.g. impossibly fast, or ridiculously slow). Remove outliers because of that. Don't use an arbitrary (which is what this is) statistical procedure over your own judgment. What if an RT of 80 ms is not 3SD away? Do you keep it? It's not physically possible to be actually based on perception of a stimulus for button presses or vocal responses. With a choice task values like the 190 ms in your plot would not be believable (and that looks like it's post outlier removal). You mention several seconds must be outliers, then remove them for that reason. They must reflect processes other than a response to the stimulus. If you have an accuracy measure you can use those to guide outlier removal. Perhaps accuracy is very low under a certain RT or drops off past a different RT.

You are not allowed to use all of the degrees of freedom in all of your measurements in single level ANOVA because this is a repeated measures design. You must aggregate your numbers so that each subject produces 4 numbers, 1 value in each condition.

items.agg <- aggregate(TextDisplay9.RT ~ Conflicting + ContextPresent + Subject, itemsitems.cropped, mean)


If you examine these aggregate data you may find your skew problem is solved through the central limit theorem (although the n is a bit low for that). It would almost definitely be solved for the reciprocal of the RT in seconds (rate). RT is an arbitrary representation of performance. It is the time it took to complete the task. The rate would be how many tasks can be completed per second. They are both easily interpretable numbers but the latter has much better statistical properties. (do not forget that the meaning of rate is opposite of RT in that higher numbers = better performance)

And then you have to stratify your results in the repeated measures ANOVA

m <- aov(TextDisplay9.RT ~ Conflicting * ContextPresent + Error(Subject / (Conflicting * ContextPresent)), data = items.agg)
summary(m)


You'll have much less power in your study now.

You really should look at Baayen's (2008) book on studying linguistic RT data. It's very specific to your field and avoids much of the messy statistical theory while being very practically helpful.

Miller, J. (1991). Reaction time analysis with outlier exclusion: Bias varies with sample size. The Quarterly Journal of Experimental Psychology, 43A(4):907–912.

• (+1) Nice response. The first paragraph is very well put. – chl Jul 11 '12 at 12:42

The box-and-whisker plots look very skewed, and that is after removing the outliers.

My suggestion: Transform each value to its reciprocal. Now instead of assessing how long something takes, you will be assessing how fast it is. Now look at the distribution of values (before removing any outliers). My guess is that the data will be far more symmetrical, and you'll have many fewer outliers. Also my guess is that the data will be closer to Gaussian, which means an ANOVA on the transformed values will be more valid than the original ANOVA.

Also note that while the P value was small in your ANOVA for the "conflicting" factor, the effect size is tiny. The R squared is computed from the ratio of SS, so is only 111185/10253667 = 0.0108.

• I did compute an Anova for the reciprocal as well. R² in this case was 0.099, F(1,651) = 6.4282, p < 0.0115. I had a look at the density functions - the reciprocal transformed one looks "more" Gaussian. – Aleksandar Dimitrov Jul 11 '12 at 9:02
• @AleksandarDimitrov: It would be good to show the box-whisker plots here so others can see how transforming solved the problem. Were there any "outliers" in the transformed data? Or (my guess) were the outliers before simply the tail of a nongaussian distribution? Did the transformed data pass normality tests? – Harvey Motulsky Jul 11 '12 at 13:43

The lines in the boxplots are medians not means. When you have outliers the mean and medians can be very different. Did you include the outliers when fitting the model? If you excldued them what was your justification for removing them? Also if the outliers are not used in the model you get a misleading picture including them in the boxplot. The distributions look skewed, particularly the one on the far right and far left. Observations should not be arbitrarily removed for exceeding the 3 sigma limit. When the data is very nonnormal the F tests in the analysis of variance are not valid and you should consider nonparametric alternatives such as the Kruskal-Wallis test. Also keep in mind that even for a normal distribution approxiamtely 3 out of 1000 observations will fall outside the 3 sigma limit. With a sample of over 650 observations it would not be surprising to have one or two outside the 3 sigma limit.

• It seems to be common practice in the literature to remove observations higher than 2 or 3 standard deviations from the mean, that's why I did it — reading times studies tend to produce extreme outliers of several seconds (where the mean and median here are around 400ms.) The data I removed amounted to around 2% of the overall data. I'll look into the Kurskal-Wallis test, thanks! – Aleksandar Dimitrov Jul 11 '12 at 9:08
• I don't know what you mean by common practice in the literature. Among statisticians it is considered bad practice. – Michael R. Chernick Jul 11 '12 at 10:15
• Agreed that this is bad practice - outliers should be removed if you have reason to believe they "do not belong" with the rest of the data e.g. because of a data transcription error. For example, if you had a data set of heights and noticed that one person was 20ft tall, you'd probably remove that. Removing observations based only on the fact that they are 2+ standard deviations from the mean is bad practice. There should be some substance underlying a decision to delete a point as an "outlier". What you're doing is trimming the tails of the distribution, which is something entirely different. – Macro Jul 11 '12 at 12:25
• And, within the broader literature you're discussing, there are papers with strong arguments against it. – John Jul 11 '12 at 12:26
• @Macro I have done a lot of research on outliers and outlier detection and the common terminology that we use is any observation that stands out apart from the bulk of the distribution. It could be an unsual or extreme observation but not necessarily a candidate to be removewd from the sample. – Michael R. Chernick Jul 11 '12 at 13:42