Applying modified z-score for outlier elimination on some data (Iglewicz and Hoaglin, 1993), I discovered that a big proportion of the data (~10%) was outside the range abs(z)>=3.5
. Further investigation showed that the data is heavy-tailed. I assumed that the Bienaymé–Chebyshev inequality would hold for the median absolute deviation MAD, too, but obviously does not.
.sum.bool <- function(x) c('TRUE'=sum(x),'FALSE'=sum(!x),
'TRUE %'=round(sum(x)/length(x)*100,1), length=length(x))
rrn <- rnorm(10000)
rrt <- rt(10000,1)
# simplified z-score for demonstration purposes
mad.outlier <- function(x)abs(x-mean(x))/mad(x) > 3
sd.outlier <- function(x)abs(x-mean(x))/sd(x) > 3
rbind(mad.n=.sum.bool(mad.outlier(rrn)),
sd.n=.sum.bool(sd.outlier(rrn)),
mad.t=.sum.bool(mad.outlier(rrt)),
sd.t=.sum.bool(sd.outlier(rrt)))
On the heavy-tailed t-distribution with 1df, 14% of the data are outside 3 MADs
.
TRUE FALSE TRUE % length
mad.n 29 9971 0.3 10000
sd.n 29 9971 0.3 10000
mad.t 1381 8619 13.8 10000
sd.t 33 9967 0.3 10000
Can someone shed light on this property of the MAD
/z-score in the presence of heavy-tailed distributions? What are recommendations for outlier detection for heavy-tailed data?