# Outlier detection for heavy-tailed data

Applying modified z-score for outlier elimination on some data (Iglewicz and Hoaglin, 1993), I discovered that a big proportion of the data (~10%) was outside the range abs(z)>=3.5. Further investigation showed that the data is heavy-tailed. I assumed that the Bienaymé–Chebyshev inequality would hold for the median absolute deviation MAD, too, but obviously does not.

.sum.bool  <- function(x) c('TRUE'=sum(x),'FALSE'=sum(!x),
'TRUE %'=round(sum(x)/length(x)*100,1), length=length(x))

rrn <- rnorm(10000)
rrt <- rt(10000,1)

# simplified z-score for demonstration purposes
sd.outlier <- function(x)abs(x-mean(x))/sd(x) > 3

sd.n=.sum.bool(sd.outlier(rrn)),
sd.t=.sum.bool(sd.outlier(rrt)))


On the heavy-tailed t-distribution with 1df, 14% of the data are outside 3 MADs.

      TRUE FALSE TRUE % length
sd.n    29  9971    0.3  10000
sd.t    33  9967    0.3  10000


Can someone shed light on this property of the MAD/z-score in the presence of heavy-tailed distributions? What are recommendations for outlier detection for heavy-tailed data?

-

Ignoring the tails, the Gaussian and Cauchy (T-dist w/ DF=1) look pretty similar in their meaty center. The MAD only looks at the meaty center (more-or-less). The MAD estimates will be pretty similar, which will give a pretty similar range of "acceptable". The Cauchy, with it's fat tails, will violate that acceptable range more often.

I'm not sure what your intentions are with this experiment though. In my experience, most real world data would lie somewhere in the middle of the Gaussian to Cauchy spectrum. When I apply robust statistics, I really just want to focus on capping the influence of any single point. There isn't really a "correct" answer. Robust statistics are more focused on reasonable estimates that match the bulk of your data (but not all of it). In the real world, all definitions of "outlier" are subjective. You need to tune acceptance levels to fit your own needs.

-
Thanks for your answer. The data at hand is also 'somewhere in the middle of the Gaussian to Cauchy spectrum' (and less like the standard Cauchy of the example). What I am not sure, however, is if I should use a robust measure of the variation or the standard deviation to defined outlier cutoffs. For the sd I can assure that I will not disregard too many measurements as it does not just model the meaty center in heavy-tailed distributions. –  Florian Bw Aug 6 '12 at 16:30
In general, I'd always use a robust measure of scale to identify outliers. If you don't, extreme outliers tend to mask moderate outliers. I'd also recommend moving past the MAD to something more efficient like Qn or a 2-stage Tau estimate of scale if you're serious about this. Package robustbase in R provides many of these alternatives and more. –  Shea Parkes Aug 7 '12 at 11:28

You need to look at this problem in terms of what you mean by a univariate outlier. The traditional theory of univariate outlier detection is that the data are normally distributed and points extreme with respect to normal observations are outliers. That is the basis of Grubbs' test which has some optimality properties and Dixon's ratio test. As I mentioned on previous posts about outliers, my paper in the American Statistician "A Note on the Robustness of Dixon's Ratio test in Small Samples" shows that for really small sample size 3 to 5 the test maintains its significance levels for a variety of non-normal distributions. It has often been used with running samples of size 3 as a screening procedure. But there are a lot of subtleties with outlier detection including the masking problem and the issue of heavytailed distributions. I recommend that you look at the two excellent books on outliers (1) Barnett and Lewis "Outliers in Statistical Data" and (2) Douglas Hawkins monograph on outliers.

Most importantly though is how to interpret outliers once you found them. An outlier found by Grubbs' test for example could be due to a contaminated normal model or simply because the distribution is heavytailed. There are other possibilities too. Is the outlier an error, just a chance occurrence or an indication of an important change in the process? These questions require understanding of the data and cannot be answered on purely statistical grounds.

Now if I am to suppose from your question that you know that your data represent iid observations from a heavytailed distribution, then if you can assume a parametric form for that distribution you can derive or approximate the distribution of the maximum and minimum for a sample of size n and use that as the null distribution to test for an outlier on a sample. If you have to estimate the parameter(s) of the distribution then you would need to convince yourself that a given set of data is outlier free and use that data to fit the parameter(s). Then that fitted model could be used to test on an independent set of data.

-
Thank you for your reading recommendations. The outliers which I want to capture are really erroneous readings, and the sample sizes are between a couple and a few dozens (1000s of variables are in the same data set, each variable has variable number of readings!). I am a bit hesitant to model a distribution for outlier detection, but I will certainly investigate if the measurements distribute the same across the variables. –  Florian Bw Aug 6 '12 at 16:37