How to detect outliers in skewed data set?

Question

I am working on my school datamining project. Within preprocessing stage I need to remove outliers from my data set which is positively skewed (see description). I have an idea to remove all values which are larger than mean + 3 x standard deviation, but I am not sure this is a suitable technique for my case because the data set is not normally distributed. What technique should I use?

  var     n    mean      sd  median trimmed     mad  min     max   range skew kurtosis   se
1   1 41019 1668.99 1107.08 1453.68 1524.22 1026.05 10.9 5920.74 5909.84 1.18 1.33 5.47

Answer 1

Bottom line is that the decision to remove data from your dataset is a subject-matter decision, not a statistical decision. The statistics help you to identify outliers given what you believe about the dataset.

A very readable applied treatment of outliers is given in

• B. Iglewicz and D. C. Hoaglin, How to Detect and Handle Outliers (Milwaukee: ASQC Press) 1993.

A more advanced and detailed treatment is given in

• V. Barnett and T. Lewis, Outliers in Statistical Data (New York: John Wiley and Sons) 1994.

Answer 2

Bacon answered this question a few centuries ago in Novum Organum. To paraphrase: To do science is to search for repeated patterns. To detect anomalies is to identify values that do not follow repeated patterns. "For whoever knows the ways of Nature will more" easily notice her deviations and, on the other hand, whoever knows her deviations "will more accurately describe her ways." One learns the rules by observing when the current rules fail.

In summary, build a model for your data using both user-specified variables and variables that can be suggested by residual diagnostic checking (in time series that would be level shifts, local time time trends, seasonal pulses, changes in parameters, or changes in variance). After forming a useful model, evaluate/scrutinize the residuals for unusual patterns; perhaps activity before and after known events. In this way you can iterate to identifying anomalous data.

Answer 3

Flagging outlier is not a subject-matter decision but a statistical one. Outliers have a precise, objective definition: they are observations that do not follow the pattern of the majority of the data. Such observations need to be set apart at the onset of any analysis simply because their distance from the bulk of the data ensures that they will exert a disproportionate pull on any model fitted by maximum likelihood.

Furthermore, detecting outliers is a statistical procedure with a well defined objective and whose efficacy can be measured. It is also important to point out that no matter how they are identified (whether according to an algorithm or simply through faith in someone else's wild guesses) the outlyingness of a group of suspect observations can be assessed simply by measuring their influence on a non-robust fit: outliers are by definition observations that have an abnormal leverage (or 'pull') over the coefficients obtained from an LS/ML fit. In other words, outliers are observations whose removal from the sample should severely impact the LS/ML fit. I have added more explanation of this in my answer to a related question.

In any case, the rule you cite for detecting outliers is flawed. To see why, just notice that the sum of the squared z-scores always sum to a constant (n-1), regardless of whether your data contains outliers or not. For the precise problem you have I explained at length in previous answer how adjusted boxplots could be used to identify outliers when the observations of interest are suspected to have a skewed distribution.

As pointed out by Placidia I suspect you are not providing us with all the elements for it is indeed strange to be doing data mining on univariate datasets.

Regardless, I advise you to have a look at a modern book on outlier detection methods. I warmly recommend Maronna R. A., Martin R. D. and Yohai V. J. (2006). Robust Statistics: Theory and Methods. Wiley, New York.

Answer 4

From a scientific point of view you only remove an outlier if: it's a data entry error, measurement error, or scientifically impossible. Otherwise don't remove an outlier. Try using boxplot, cleveland plot, conditional cleveland plot, and track the outliers. If you still can't justify them then try transforming your variable.

Answer 5

In addition to the other responses, don't forget to state how you identified the outliers and to provide individual details of them in your report.

Answer 6

If one cannot prespecify conditions upon which data should be excluded from an analysis independent of observed results, then the process of removing outliers via visual inspection or some data-driven automated process introduces great bias into any analysis.

This is because you greatly inflate the type I error by admitting yourself to excluding points which do not drive a trend you would otherwise not believe to exist. In the analysis of data, there are always points which heavily drive an analysis, but when a scientific question (rather than data) drive an analysis, exclusion criteria automatically restrict observations to a sample which we believe to be reflective of the population. For instance, in a sample of 100, if I observe 1 "outlier", I know that "outlier" is representative of 1/100 of the population I'm interested in.

A primary analysis would a priori specify exclusion criteria in such a way that such a point would be reliably excluded from any subsequent attempt to recreate our analysis in independent data. For instance, if we specify that "people with household incomes of 500,000 USD or more were excluded", investigators could replicate your analysis at any sample size and each household satisfying that criterion would be excluded. Exclusion criteria based on the mean and standard deviation of observed household incomes would not reliably exclude the same households from the primary analysis each time. This is another source of bias.