# Outlier detection for heavy-tailed data

Applying modified z-score for outlier elimination on some data (Iglewicz and Hoaglin, 1993), I discovered that a big proportion of the data (~10%) was outside the range abs(z)>=3.5. Further investigation showed that the data is heavy-tailed. I assumed that the Bienaymé–Chebyshev inequality would hold for the median absolute deviation MAD, too, but obviously does not.

.sum.bool  <- function(x) c('TRUE'=sum(x),'FALSE'=sum(!x),
'TRUE %'=round(sum(x)/length(x)*100,1), length=length(x))

rrn <- rnorm(10000)
rrt <- rt(10000,1)

# simplified z-score for demonstration purposes
sd.outlier <- function(x)abs(x-mean(x))/sd(x) > 3

sd.n=.sum.bool(sd.outlier(rrn)),
sd.t=.sum.bool(sd.outlier(rrt)))


On the heavy-tailed t-distribution with 1df, 14% of the data are outside 3 MADs.

      TRUE FALSE TRUE % length
sd.n    29  9971    0.3  10000
sd.t    33  9967    0.3  10000


Can someone shed light on this property of the MAD/z-score in the presence of heavy-tailed distributions? What are recommendations for outlier detection for heavy-tailed data?

-

Ignoring the tails, the Gaussian and Cauchy (T-dist w/ DF=1) look pretty similar in their meaty center. The MAD only looks at the meaty center (more-or-less). The MAD estimates will be pretty similar, which will give a pretty similar range of "acceptable". The Cauchy, with it's fat tails, will violate that acceptable range more often.

I'm not sure what your intentions are with this experiment though. In my experience, most real world data would lie somewhere in the middle of the Gaussian to Cauchy spectrum. When I apply robust statistics, I really just want to focus on capping the influence of any single point. There isn't really a "correct" answer. Robust statistics are more focused on reasonable estimates that match the bulk of your data (but not all of it). In the real world, all definitions of "outlier" are subjective. You need to tune acceptance levels to fit your own needs.

-
Thanks for your answer. The data at hand is also 'somewhere in the middle of the Gaussian to Cauchy spectrum' (and less like the standard Cauchy of the example). What I am not sure, however, is if I should use a robust measure of the variation or the standard deviation to defined outlier cutoffs. For the sd I can assure that I will not disregard too many measurements as it does not just model the meaty center in heavy-tailed distributions. –  Florian Bw Aug 6 '12 at 16:30
In general, I'd always use a robust measure of scale to identify outliers. If you don't, extreme outliers tend to mask moderate outliers. I'd also recommend moving past the MAD to something more efficient like Qn or a 2-stage Tau estimate of scale if you're serious about this. Package robustbase in R provides many of these alternatives and more. –  Shea Parkes Aug 7 '12 at 11:28