0
$\begingroup$

Some people seem to frown on removing outliers. But I've also read many times elsewhere that ANOVAs are sensitive to outliers and you must remove them.

I'm running a 2 x 2 repeated measures within subjects ANOVA and there are a number of outliers on each level of the IVs. I'm not sure what to do. You may suggest using a different test, but this isn't really an option. Is there anything I can do if I use this test? The definition of outlier I'm using here is the one used by SPSS which I believe is this:

3rd quartile + 1.5*interquartile range

1st quartile – 1.5*interquartile range

Now, just to see what it would look like, I removed all outliers that adhered to this, and looked at my boxplots again. Presumably because the interquartile range itself has now changed after removing cases, more outliers simply appeared. Not good. I can't just keep removing them... can I?

So yeah, I'm stuck. Any ideas?

$\endgroup$
9
  • 1
    $\begingroup$ That ANOVA is sensitive to extreme observations is why it is important to keep them. $\endgroup$
    – Dave
    Commented Dec 20, 2021 at 4:18
  • $\begingroup$ @Dave but you're supposed to check assumptions of ANOVA before you do them, right? See here: statistics.laerd.com/spss-tutorials/… The fact mine has violated one of these key assumptions... isn't that a problem? My analysis will no longer be valid, right? So I'm not sure what to do. I have to report the main effects, simple effects, etc. But then presumably I'd be all 'but yeah all this is nonsense, because of the outliers, so...' which doesn't sound ideal. $\endgroup$ Commented Dec 20, 2021 at 4:35
  • 2
    $\begingroup$ Then you change your modeling approach to reflect the reality of the data, rather than changing your data to reflect the assumptions of a mathematical procedure. $\endgroup$
    – Dave
    Commented Dec 20, 2021 at 4:38
  • 1
    $\begingroup$ With more information about your proposed experimental design, the type of data, and your objectives, it might be easier to give more relevant advice . [(i) "$2\times 2$ repeated measures" could mean several different things. (ii) How do the outliers arise? What do they mean? What do you most want to know from your data?] An ANOVA on overall ranks would not ensure exact normality of residuals, but it would lessen the effect of outliers without censoring them. A regression approach as suggested by @Dave might work. It might be possible to find an appropriate metric for a permutation test, Etc. $\endgroup$
    – BruceET
    Commented Dec 20, 2021 at 6:08
  • 1
    $\begingroup$ @BruceET Hmm, well, it's a 2 x 2 Stroop task measuring reaction time as the dependent variable, with posture and congruence as dependent variables, each with two levels. I'm running it in SPSS, a standard within subjects repeated measures ANOVA. When I generate boxplots to assess outliers of reaction time, I get like 4 or 5 out of around 200. I've read that ANOVAs are robust against normality violations, but not outliers. I want to know whether there are effects of posture and congruence on reaction time, mainly. But will I be able to determine any main effects with these outliers? $\endgroup$ Commented Dec 21, 2021 at 2:42

2 Answers 2

1
$\begingroup$

One difficulty (out of several) with far outliers in what ought to be normal data is that the null hypothesis (no differences) may be rejected too often, leading to false discovery.

For simplicity, I will illustrate with pooled 2-sample t tests--instead of ANOVAs. Consider the following fictitious data comparing two samples of size 20 from the same exponential distribution. There is no difference between the two populations, so a no test should reject the null hypothesis. We look first at a pooled 2-sample t test.

set.seed(2021)
x1 = rexp(20, 1)
x2 = rexp(20, 1)
x = c(x1,x2)
g = rep(1:2, each=20)
boxplot(x~g, horizontal=T)

enter image description here

Nevertheless, the pooled 2-sample t test is (narrowly) significant at the 5% level.

t.test(x~g, var.eq=T)

        Two Sample t-test

data:  x by g
t = -2.0987, df = 38, p-value = 0.04254
alternative hypothesis: 
 true difference in means is not equal to 0
95 percent confidence interval:
 -1.02119881 -0.01840687
sample estimates:
mean in group 1 mean in group 2 
      0.4915494       1.0113522 

A simulation shows that this kind of false rejection occurs about half the time two such exponential samples are compared with a pooled 2-sample t test.

pv = replicate(10^5, t.test(c(rexp(20,1),rexp(20,1))~g, 
var.eq=T)$p.val)
mean(pv <= 0.5)
[1] 0.51002

By contrast, if we take ranks of the combined data, the ranks will run from 1 to 40, so outliers are not likely. Yet, the relative standing of the values in the two samples is preserved. Consequently, the pooled 2-sample t test (correctly) does not reject.

boxplot(rank(x)~g, horizontal=T)

enter image description here

t.test(rank(x)~g, var.eq=T)

        Two Sample t-test

data:  rank(x) by g
t = -1.5413, df = 38, p-value = 0.1315
alternative hypothesis: 
 true difference in means is not equal to 0
95 percent confidence interval:
-12.955283   1.755283
sample estimates:
mean in group 1 mean in group 2 
           17.7            23.3 

For ranked data, a simulation shows that the pooled 2-sample t test seldom rejects.

pv.r = replicate(105, t.test(rank(c(rexp(20,1),rexp(20,1)))~g, 
                 var.eq=T)$p.val)
mean(pv.r <= .05)
[1] 0.01904762

In this case, the true rejection rate when there is no difference between the two populations is about 2%. Granted, about 5% would be better, but doing the pooled 2-sample t test on ranked data is better than ignoring the skewness of exponential data and resuting outliers.

$\endgroup$
0
$\begingroup$

(+1) to BruceET. In Bruce's example the data generative process is clearly not normal. The performance of the two-sample t-test on the raw data might be improved by incorporating a log link function. Additionally, the score and likelihood ratio test would have improved performance compared to a t- or Wald test with an identity link. Here is a related link if your data is very clearly non-normal.

I generally lean towards not removing outliers or removing them as part of a sensitivity analysis if the outliers are a result of improper data collection. For a given sensitivity analysis I would set the outliers to missing and consider a missing data assumption (the simplest being an ignorable missing data mechanism). To pressure test the missing data assumption several sensitivity analyses can be performed, each under a different missing data assumption.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.