People often talk about dealing with outliers in statistics. The thing that bothers me about this is that, as far as I can tell, the definition of an outlier is completely subjective. For example, if the true distribution of some random variable is very heavy-tailed or bimodal, any standard visualization or summary statistic for detecting outliers will incorrectly remove parts of the distribution you want to sample from. What is a rigorous definition of an outlier, if one exists, and how can outliers be dealt with without introducing unreasonable amounts of subjectivity into an analysis?
$\begingroup$
$\endgroup$
4
8 Answers
$\begingroup$
$\endgroup$
6
As long as your data comes from a known distribution with known properties, you can rigorously define an outlier as an event that is too unlikely to have been generated by the observed process (if you consider "too unlikely" to be non-rigorous, then all hypothesis testing is).
However, this approach is problematic on two levels: It assumes that the data comes from a known distribution with known properties, and it brings the risk that outliers are looked at as data points that were smuggled into your data set by some magical faeries.
In the absence of magical data faeries, all data comes from your experiment, and thus it is actually not possible to have outliers, just weird results. These can come from recording errors (e.g. a 400000 bedroom house for 4 dollars), systematic measurement issues (the image analysis algorithm reports huge areas if the object is too close to the border) experimental problems (sometimes, crystals precipitate out of the solution, which give very high signal), or features of your system (a cell can sometimes divide in three instead of two), but they can also be the result of a mechanism that no one's ever considered because it's rare and you're doing research, which means that some of the stuff you do is simply not known yet.
Ideally, you take the time to investigate every outlier, and only remove it from your data set once you understand why it doesn't fit your model. This is time-consuming and subjective in that the reasons are highly dependent on the experiment, but the alternative is worse: If you don't understand where the outliers came from, you have the choice between letting outliers "mess up" your results, or defining some "mathematically rigorous" approach to hide your lack of understanding. In other words, by pursuing "mathematical rigorousness" you choose between not getting a significant effect and not getting into heaven.
EDIT
If all you have is a list of numbers without knowing where they come from, you have no way of telling whether some data point is an outlier, because you can always assume a distribution where all data are inliers.
-
4$\begingroup$ Not all outliers are generated from an experiment, however. I worked with a large dataset that involved the collection of real-estate information in a region (sale price, number of bedrooms, square footage, etc), and every now and then, there would be data entry mistakes and I'd have a 400,000 bedroom house go for 4 dollars, or something nonsensical like that. I would think that part of the goal of determining an outlier is to see whether it's possible to be generated from the data, or if it was just an entry error. $\endgroup$ Feb 13, 2011 at 19:25
-
2$\begingroup$ @Christopher Aden: I'd consider that part of the experimental process. Basically, in order to be able to remove outliers, you have to understand how the data were generated, i.e. no removing outliers without a good reason. Otherwise you're just stylizing your data. I've edited my answer to reflect this a bit better. $\endgroup$– JonasFeb 13, 2011 at 19:40
-
$\begingroup$ This is perfectly reasonable, but assumes you already have a decent amount of prior knowledge about what the true distribution is. I was thinking more in terms of scenarios where you don't and it could be very heavy tailed or bimodal. $\endgroup$– dsimchaFeb 14, 2011 at 3:06
-
1$\begingroup$ @dsimcha: I don't think you can identify outliers in that case (see also my edit). $\endgroup$– JonasFeb 14, 2011 at 4:08
-
3$\begingroup$ @dsimcha - you always have prior knowledge! for how were the data given to you? you always always know that much. data doesn't magically just show up. and you can always make tentative assumptions. "outliers" based on these assumptions basically give you a clue that something in your assumptions is wrong. by studying the "outlier" (which is always relative) you can improve your model. $\endgroup$ Feb 16, 2011 at 12:50
$\begingroup$
$\endgroup$
2
You are correct that removing outliers can look like a subjective exercise but that doesn't mean that it's wrong. The compulsive need to always have a rigorous mathematical reason for every decision regarding your data analysis is often just a thin veil of artificial rigour over what turns out to be a subjective exercise anyway. This is especially true if you want to apply the same mathematical justification to every situation you come across. (If there were bulletproof clear mathematical rules for everything then you wouldn't need a statistician.)
For example, in your long tail distribution situation, there's no guaranteed method to just decide from the numbers whether you've got one underlying distribution of interest with outliers or two underlying distributions of interest with outliers being part of only one of them. Or, heaven forbid, just the actual distribution of data.
The more data you collect, the more you get into the low probability regions of a distribution. If you collect 20 samples it's very unlikely you'l get a value with a z-score of 3.5. If you collect 10,000 samples it's very likely you'll get one and it's a natural part of the distribution. Given the above, how do you decide just because something is extreme to exclude it?
Selecting the best methods in general for analysis is often subjective. Whether it's unreasonably subjective depends on the explanation for the decision and on the outlier.
-
$\begingroup$ +1 Barnett and Lewis, who wrote the book on outliers, state "an outlier in a set of data [is] an observation (or subset of observations) which appears to be inconsistent with the remainder of that set of data" [at p. 7]. They continue, "It is a matter of subjective judgement on the part of the observer whether or not some observation ... is picked out for scrutiny. ... What characterizes the 'outlier' is its impact on the observer ... ." $\endgroup$– whuber ♦Aug 27, 2013 at 21:41
-
$\begingroup$ "the book" is slightly ambiguous here. I'd consider Barnett and Lewis the leading monograph, but it's not the only book on outliers. amazon.com/Outlier-Analysis-Charu-C-Aggarwal/dp/1461463955 is recent. There is also an older book by D.M. Hawkins. $\endgroup$– Nick CoxAug 28, 2013 at 8:19
$\begingroup$
$\endgroup$
2
I don't think it is possible to define an outlier without assuming a model of the underlying process giving rise to the data. Without such a model we have no frame of reference to decide whether the data are anomalous or "wrong". The definition of an outlier that I have found useful is that an outlier is an observation (or observations) that cannot be reconciled to a model that otherwise performs well.
answered Feb 14, 2011 at 16:36
-
2$\begingroup$ Hmm...In his EDA text, John Tukey specifically defined outliers without using any models at all. $\endgroup$– whuber ♦Mar 8, 2011 at 6:59
-
7$\begingroup$ You can define outliers without a model, but I have found such definitions to be unhelpful. BTW, by model, I don't necessarily mean a statistical model that has been explicitly fit to the data. Any definition of an outlier requires you to make some assumption about what sort of values you expect to see, and what sort of values you don't expect to see. I think it is better if these assumptions (i.e. the model) are made explicit. There is also the point that in EDA, you are exploring the data, your definition of an outlier may be very different for EDA than for fitting a final model. $\endgroup$ Mar 9, 2011 at 11:09
$\begingroup$
$\endgroup$
7
There are many excellent answers here. However, I want to point out that two questions are being confused. The first is, 'what is an outlier?', and more specifically to give a "rigorous definition" of such. This is simple:
An outlier is a data point that comes from a different population / distribution / data generating process than the one you intended to study / the rest of your data.
The second question is 'how do I know / detect that a data point is an outlier?' Unfortunately, this is very difficult. However, the answers given here (which really are very good, and which I can't improve upon) will be quite helpful with that task.
answered Dec 23, 2011 at 23:55
-
1$\begingroup$ This is a thought-provoking answer. So, suppose I generate $99$ iid values from a Normal$(0,1)$ distribution--they are likely to span a range from around $-2.5$ to $2.5$--and generate one more value from a Normal$(4,1)$ distribution and it happens to equal $2$ (for which there's about a $1$ in $40$ chance). It's highly unlikely that extra $2$ would be determined to be an outlier. Do you claim that it really is? Your quotation makes me think so, but I don't see how this can be made practically operational. $\endgroup$– whuber ♦Jul 10, 2012 at 21:25
-
1$\begingroup$ @whuber, yes. I say that it is an outlier, although you would never notice it (which, I suspect, is what you mean by practically operational). $\endgroup$ Jul 10, 2012 at 21:35
-
1$\begingroup$ I appreciate the distinction you're making. I just wanted to point out the sharp contrast between your definition and most of the other definitions or descriptions of outliers in this thread. Yours does not seem like it could lead to satisfactory practical procedures: you would always have to accept that a huge part of your dataset might be "outlying" but without having any way to detect or resolve that. $\endgroup$– whuber ♦Jul 10, 2012 at 21:40
-
1$\begingroup$ What you call an "outlier" here is often referred to as a "contaminant." An observation might or might not be a contaminant--this is objectively either true or false, but often impossible to ascertain--whereas whether an observation (or set of observations) is outlying is a matter of how the data are examined and therefore is a subjective distinction. (That does not mean it's a useless or vague one, however!) It looks advisable to maintain this useful distinction between outliers and contamination rather than conflating the two concepts. $\endgroup$– whuber ♦Aug 27, 2013 at 21:49
-
2$\begingroup$ @whuber, you're right. Some make this distinction, but many aren't clear about these ideas. My position is that there is no meaningful reality of "outlier" other than contaminant. Nonetheless, people should also / instead think about the issue as being concerned about point(s) if your results are driven by them alone (whether they are 'real' or not), & thus your results are very fragile. In short, there is no reason to be concerned about point(s) that are from your population & aren't uniquely driving your results; once you've dealt w/ those 2 issues, there is nothing left to "outlier". $\endgroup$ Aug 27, 2013 at 22:14
$\begingroup$
$\endgroup$
Definition 1: As already mentioned, an outlier in a group of data reflecting the same process (say process A) is an observation (or a set of observations) that is unlikely to be a result of process A.
This definition certainly involves an estimation of the likelihood function of the process A (hence a model) and setting what unlikely means (i.e. deciding where to stop...). This definition is at the root of the answer I gave here. It is more related to ideas of hypothesis testing of significance or goodness of fit.
Definition 2 An outlier is an observation $x$ in a group of observations $G$ such that when modeling the group of observation with a given model the accuracy is higher if $x$ is removed and treated separately (with a mixture, in the spirit of what I mention here).
This definition involves a "given model" and a measure of accuracy. I think this definition is more from the practical side and is more at the origin of outliers. At the Origin, outlier detection was a tool for robust statistics.
Obviously these definitions can be made very similar if you understand that calculating likelihood in the first definition involves modeling and calculation of a score :)
answered Feb 13, 2011 at 19:30
$\begingroup$
$\endgroup$
7
An outlier is a data point that is inconvenient to me, given my current understanding of the process that generates this data.
I believe this definition is as rigorous as can be made.
-
$\begingroup$ Contrast this to John Tukey's definition (he used the term "outside"): "When we look at some batches of values, we see certain values as apparently straying out far beyond the others. ... It is convenient to have a rule of thumb that picks out certain values as "outside" ..." Later he summarizes this as "...identification of individual values that may be unusual." [EDA, chapter 2]. He emphasizes throughout the book that we are describing data rather than pretending to "understand a process," and that multiple valid descriptions are always possible. $\endgroup$– whuber ♦Aug 27, 2013 at 21:22
-
2$\begingroup$ Similarly, "Outliers are sample values that cause surprise in relation to the majority of the sample" (W.N. Venables and B.D. Ripley. 2002. Modern applied statistics with S. New York: Springer, p.119). However, surprise is in the mind of the beholder and is dependent on some tacit or explicit model of the data. There may be another model under which the outlier is not surprising at all, say, the data really are lognormal or gamma rather than normal. $\endgroup$– Nick CoxAug 27, 2013 at 21:33
-
$\begingroup$ @Nick That is consistent with Barnett and Lewis, whom I quote in a comment to John's answer. $\endgroup$– whuber ♦Aug 27, 2013 at 21:44
-
1$\begingroup$ @whuber: You say "Contrast this", which I think means you disagree, but I'm not sure. I'd argue that model-formation -- implicit and naive, perhaps -- is why we see patterns in data, or the man in the moon, or outliers. The model may have no physics/chemistry/economic basis, but we have hypothesized a model. Otherwise, there is no surprise, there is no "outside". $\endgroup$– WayneAug 27, 2013 at 21:48
-
$\begingroup$ Tukey is insistent that in describing data we are not necessarily modeling them. It's fair to extend your definition of "model" to include data description, but then the term becomes almost too general to be useful. From Tukey's point of view (as I interpret it, of course), there is no concern about loss of face nor is there any question of convenience or not. Thus, although I respect your motivation, I think your attitude (as reflected in "face-saving" and "inconvenient") is less constructive than other approaches to this question. $\endgroup$– whuber ♦Aug 27, 2013 at 21:54
$\begingroup$
$\endgroup$
define an outlier as a member of that minimal set of elements which must be removed from a datasetof size n in order to assure 100% compliance with RUM tests conducted at 95% confidence level on all (2^n -1) unique subsets of the data. See Karian and Dudewicz text on fitting data to pdfs using R(Sept 2010) for definition of the RUM test.
$\begingroup$
$\endgroup$
1
Outliers are important only in the frequentist realm. If a single datapoint adds bias to your model which is defined by an underlying distribution predeterimined by your theory, then it is an outlier for that model. The subjectivity lies in the fact that if your theory posits a different model, then you can have a different set of points as outliers.
-
1$\begingroup$ Are you claiming outliers are unimportant in Bayesian data analysis? $\endgroup$– whuber ♦Mar 8, 2011 at 7:00
rigorous definition of an outlierwhen you'll be able to defineunreasonable amounts of subjectivityobjective manner ;-), Thanks $\endgroup$