3
$\begingroup$

I am working on my school datamining project. Within preprocessing stage I need to remove outliers from my data set which is positively skewed (see description). I have an idea to remove all values which are larger than mean + 3 x standard deviation, but I am not sure this is a suitable technique for my case because the data set is not normally distributed. What technique should I use?

  var     n    mean      sd  median trimmed     mad  min     max   range skew kurtosis   se
1   1 41019 1668.99 1107.08 1453.68 1524.22 1026.05 10.9 5920.74 5909.84 1.18 1.33 5.47
$\endgroup$
  • 4
    $\begingroup$ Why do you want to remove outliers? Doing this by any automatic method is usually a bad idea, although it depends on what you are trying to do. $\endgroup$ – Peter Flom - Reinstate Monica Feb 24 '13 at 13:18
  • $\begingroup$ @PeterFlom: can you suggest a non-automatic method for removing outliers? $\endgroup$ – user603 Feb 24 '13 at 15:28
  • 5
    $\begingroup$ I believe Peter Flom is referring to getting to know your data and how it might be generated and what kinds of limits might make sense. For example, if you have a variable that is a person's age, you know it can't be negative and reasonably won't be greater than, say, 120. Automatically throwing out everything greater than 3 SD away from the mean could well throw out some 90-year-olds who snuck into your dataset, yet also keep some people with an age of -4. $\endgroup$ – Wayne Feb 24 '13 at 16:11
  • $\begingroup$ @Wayne: Where is this throwing out everything greater than 3 SD rule coming from? $\endgroup$ – user603 Feb 24 '13 at 16:50
  • 1
    $\begingroup$ @user603: from the original posting: "I have an idea to remove all values which are larger than mean + 3 x standard deviation". Which would correspond to a 99.72% CI, I believe. In case there was confusion, I was showing how a "throw everything out over X SD from the mean" rule could well lead to mistakes in both directions: throwing out things that should be kept, and keeping things that should be thrown out. $\endgroup$ – Wayne Feb 24 '13 at 17:24
4
$\begingroup$

Bottom line is that the decision to remove data from your dataset is a subject-matter decision, not a statistical decision. The statistics help you to identify outliers given what you believe about the dataset.

A very readable applied treatment of outliers is given in

A more advanced and detailed treatment is given in

$\endgroup$
  • $\begingroup$ I tweaked this to make it slightly more readable & easier to find out more, I hope you don't mind. $\endgroup$ – gung - Reinstate Monica Feb 24 '13 at 16:51
2
$\begingroup$

Flagging outlier is not a subject-matter decision but a statistical one. Outliers have a precise, objective definition: they are observations that do not follow the pattern of the majority of the data. Such observations need to be set apart at the onset of any analysis simply because their distance from the bulk of the data ensures that they will exert a disproportionate pull on any model fitted by maximum likelihood.

Furthermore, detecting outliers is a statistical procedure with a well defined objective and whose efficacy can be measured. It is also important to point out that no matter how they are identified (whether according to an algorithm or simply through faith in someone else's wild guesses) the outlyingness of a group of suspect observations can be assessed simply by measuring their influence on a non-robust fit: outliers are by definition observations that have an abnormal leverage (or 'pull') over the coefficients obtained from an LS/ML fit. In other words, outliers are observations whose removal from the sample should severely impact the LS/ML fit. I have added more explanation of this in my answer to a related question.

In any case, the rule you cite for detecting outliers is flawed. To see why, just notice that the sum of the squared z-scores always sum to a constant (n-1), regardless of whether your data contains outliers or not. For the precise problem you have I explained at length in previous answer how adjusted boxplots could be used to identify outliers when the observations of interest are suspected to have a skewed distribution.

As pointed out by Placidia I suspect you are not providing us with all the elements for it is indeed strange to be doing data mining on univariate datasets.

Regardless, I advise you to have a look at a modern book on outlier detection methods. I warmly recommend Maronna R. A., Martin R. D. and Yohai V. J. (2006). Robust Statistics: Theory and Methods. Wiley, New York.

$\endgroup$
2
$\begingroup$

Bacon answered this question a few centuries ago in Novum Organum. To paraphrase: To do science is to search for repeated patterns. To detect anomalies is to identify values that do not follow repeated patterns. "For whoever knows the ways of Nature will more" easily notice her deviations and, on the other hand, whoever knows her deviations "will more accurately describe her ways." One learns the rules by observing when the current rules fail.

In summary, build a model for your data using both user-specified variables and variables that can be suggested by residual diagnostic checking (in time series that would be level shifts, local time time trends, seasonal pulses, changes in parameters, or changes in variance). After forming a useful model, evaluate/scrutinize the residuals for unusual patterns; perhaps activity before and after known events. In this way you can iterate to identifying anomalous data.

$\endgroup$
  • 1
    $\begingroup$ +1 Good advice--and it's interesting to see such explicit connections made between modern (statistical) practice and the earliest thoughts about scientific method. $\endgroup$ – whuber Feb 26 '13 at 13:57
1
$\begingroup$

From a scientific point of view you only remove an outlier if: it's a data entry error, measurement error, or scientifically impossible. Otherwise don't remove an outlier. Try using boxplot, cleveland plot, conditional cleveland plot, and track the outliers. If you still can't justify them then try transforming your variable.

$\endgroup$
  • $\begingroup$ Even measurement errors, or some of the other examples, are not necessarily indicative of a protocol to remove observations from a primary analysis population. For instance, if in a clinical trial for HIV immunosuppression, there is an inferior assay for measuring viral load which gives poorly calibrated results, we should anticipate that medical experts will be using the same technology and will be unable to discern such outliers through comparison to a reference sample. $\endgroup$ – AdamO Feb 25 '13 at 3:55
  • $\begingroup$ @shadi: i have difficulty understanding what you mean by 'track the outliers'. $\endgroup$ – user603 Feb 26 '13 at 13:52
1
$\begingroup$

In addition to the other responses, don't forget to state how you identified the outliers and to provide individual details of them in your report.

$\endgroup$
  • $\begingroup$ should probably be a comment, not an answer. $\endgroup$ – user603 Feb 25 '13 at 18:23
  • 3
    $\begingroup$ Yes, but worth mentioning and at the time of posting, I didn't have authority to make comments $\endgroup$ – Robert Jones Feb 25 '13 at 19:00
0
$\begingroup$

If one cannot prespecify conditions upon which data should be excluded from an analysis independent of observed results, then the process of removing outliers via visual inspection or some data-driven automated process introduces great bias into any analysis.

This is because you greatly inflate the type I error by admitting yourself to excluding points which do not drive a trend you would otherwise not believe to exist. In the analysis of data, there are always points which heavily drive an analysis, but when a scientific question (rather than data) drive an analysis, exclusion criteria automatically restrict observations to a sample which we believe to be reflective of the population. For instance, in a sample of 100, if I observe 1 "outlier", I know that "outlier" is representative of 1/100 of the population I'm interested in.

A primary analysis would a priori specify exclusion criteria in such a way that such a point would be reliably excluded from any subsequent attempt to recreate our analysis in independent data. For instance, if we specify that "people with household incomes of 500,000 USD or more were excluded", investigators could replicate your analysis at any sample size and each household satisfying that criterion would be excluded. Exclusion criteria based on the mean and standard deviation of observed household incomes would not reliably exclude the same households from the primary analysis each time. This is another source of bias.

$\endgroup$
  • $\begingroup$ just to make this clear, are you claiming that trimmed estimators are biased and/or inconsistent? $\endgroup$ – user603 Feb 25 '13 at 7:15
  • 1
    $\begingroup$ Yes, except under certain assumptions. In a linear regression model, this is the assumption of linearity, in a Cox model, proportion hazards. With linear regression, you can make inference about correlation and first-order trend in curvilinear bivariate relationships. Removing high influence/high leverage points changes the scientific question significantly and interpreting results as they would apply to untrimmed parameters, an "unconditional population", is wrong. $\endgroup$ – AdamO Feb 25 '13 at 16:48
  • $\begingroup$ you are wrong. Quoting from the wiki page "a robust statistic is resistant to errors in the results, produced by deviations from assumptions....This means that if the assumptions are only approximately met, the robust estimator will still have a reasonable efficiency, and reasonably small bias, as well as being asymptotically unbiased, meaning having a bias tending towards 0 as the sample size tends towards infinity." In fact in many cases we can even explicitly derive their Fisher consitency factors! $\endgroup$ – user603 Feb 25 '13 at 17:16
  • $\begingroup$ If you have any proof to the contrary e.g. a proof, that any of the estimator with a trimmed influence curve --and there are many out there: (lts, lms, mcd, mve, Qn, Sn, tau-scale,mad, M-estimator,....), that they are biased, please put up. $\endgroup$ – user603 Feb 26 '13 at 7:03
  • 1
    $\begingroup$ The main point of this reply is well worth bearing in mind: unprincipled or undocumented or incautious treatment of outliers (or of other issues of data "cleaning" and preparation) can introduce bias in the results. With @user603, though, I am a little disturbed to see this point pressed perhaps beyond breaking by assertions about bias in estimators specifically designed to reduce or eliminate bias. I suspect we might all agree those assertions could actually be correct if the scope of (mis-)application ("certain assumptions") were more clearly described. $\endgroup$ – whuber Feb 26 '13 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.