Outlier detection for heavy-tailed data

Question

Applying modified z-score for outlier elimination on some data (Iglewicz and Hoaglin, 1993), I discovered that a big proportion of the data (~10%) was outside the range abs(z)>=3.5. Further investigation showed that the data is heavy-tailed. I assumed that the Bienaymé–Chebyshev inequality would hold for the median absolute deviation MAD, too, but obviously does not.

.sum.bool  <- function(x) c('TRUE'=sum(x),'FALSE'=sum(!x),
                            'TRUE %'=round(sum(x)/length(x)*100,1), length=length(x))

rrn <- rnorm(10000)
rrt <- rt(10000,1)

# simplified z-score for demonstration purposes
mad.outlier <- function(x)abs(x-mean(x))/mad(x) > 3
sd.outlier <- function(x)abs(x-mean(x))/sd(x) > 3

rbind(mad.n=.sum.bool(mad.outlier(rrn)),
      sd.n=.sum.bool(sd.outlier(rrn)),
      mad.t=.sum.bool(mad.outlier(rrt)),
      sd.t=.sum.bool(sd.outlier(rrt)))

On the heavy-tailed t-distribution with 1df, 14% of the data are outside 3 MADs.

      TRUE FALSE TRUE % length
mad.n   29  9971    0.3  10000
sd.n    29  9971    0.3  10000
mad.t 1381  8619   13.8  10000
sd.t    33  9967    0.3  10000

Can someone shed light on this property of the MAD/z-score in the presence of heavy-tailed distributions? What are recommendations for outlier detection for heavy-tailed data?

Answer 1

Ignoring the tails, the Gaussian and Cauchy (T-dist w/ DF=1) look pretty similar in their meaty center. The MAD only looks at the meaty center (more-or-less). The MAD estimates will be pretty similar, which will give a pretty similar range of "acceptable". The Cauchy, with it's fat tails, will violate that acceptable range more often.

I'm not sure what your intentions are with this experiment though. In my experience, most real world data would lie somewhere in the middle of the Gaussian to Cauchy spectrum. When I apply robust statistics, I really just want to focus on capping the influence of any single point. There isn't really a "correct" answer. Robust statistics are more focused on reasonable estimates that match the bulk of your data (but not all of it). In the real world, all definitions of "outlier" are subjective. You need to tune acceptance levels to fit your own needs.

Answer 2

You need to look at this problem in terms of what you mean by a univariate outlier. The traditional theory of univariate outlier detection is that the data are normally distributed and points extreme with respect to normal observations are outliers. That is the basis of Grubbs' test which has some optimality properties and Dixon's ratio test. As I mentioned on previous posts about outliers, my paper in the American Statistician "A Note on the Robustness of Dixon's Ratio test in Small Samples" shows that for really small sample size 3 to 5 the test maintains its significance levels for a variety of non-normal distributions. It has often been used with running samples of size 3 as a screening procedure. But there are a lot of subtleties with outlier detection including the masking problem and the issue of heavytailed distributions. I recommend that you look at the two excellent books on outliers (1) Barnett and Lewis "Outliers in Statistical Data" and (2) Douglas Hawkins monograph on outliers.

Most importantly though is how to interpret outliers once you found them. An outlier found by Grubbs' test for example could be due to a contaminated normal model or simply because the distribution is heavytailed. There are other possibilities too. Is the outlier an error, just a chance occurrence or an indication of an important change in the process? These questions require understanding of the data and cannot be answered on purely statistical grounds.

Now if I am to suppose from your question that you know that your data represent iid observations from a heavytailed distribution, then if you can assume a parametric form for that distribution you can derive or approximate the distribution of the maximum and minimum for a sample of size n and use that as the null distribution to test for an outlier on a sample. If you have to estimate the parameter(s) of the distribution then you would need to convince yourself that a given set of data is outlier free and use that data to fit the parameter(s). Then that fitted model could be used to test on an independent set of data.