# Rigorous definition of an outlier?

People often talk about dealing with outliers in statistics. The thing that bothers me about this is that, as far as I can tell, the definition of an outlier is completely subjective. For example, if the true distribution of some random variable is very heavy-tailed or bimodal, any standard visualization or summary statistic for detecting outliers will incorrectly remove parts of the distribution you want to sample from. What is a rigorous definition of an outlier, if one exists, and how can outliers be dealt with without introducing unreasonable amounts of subjectivity into an analysis?

-
If you want to know for a specific distribution then ask about your example. It will be different for different situations. –  John Feb 13 '11 at 15:55
Well, I would expect that you will have a rigorous definition of an outlier when you'll be able to define unreasonable amounts of subjectivity objective manner ;-), Thanks –  eat Feb 13 '11 at 16:00
But the definition can vary by underlying distribution and situation. I could say ± 1.5 IQR, or 3 SD, or some such. But I could take a totally different approach if I have two kinds of measures, say reaction time and accuracy. I can say RT's conditioned on a level of accuracy. They can all be good and mathematically rigorous and have different applications and meanings. –  John Feb 13 '11 at 16:08
There are MANY rigorous definitions of outlier. But the choice among those can seem arbitrary. But I think this is part of the misconception that statistics is a subject in which each problem has one correct answer. –  Peter Flom Mar 8 '11 at 1:55

As long as your data comes from a known distribution with known properties, you can rigorously define an outlier as an event that is too unlikely to have been generated by the observed process (if you consider "too unlikely" to be non-rigorous, then all hypothesis testing is).

However, this approach is problematic on two levels: It assumes that the data comes from a known distribution with known properties, and it brings the risk that outliers are looked at as data points that were smuggled into your data set by some magical faeries.

In the absence of magical data faeries, all data comes from your experiment, and thus it is actually not possible to have outliers, just weird results. These can come from recording errors (e.g. a 400000 bedroom house for 4 dollars), systematic measurement issues (the image analysis algorithm reports huge areas if the object is too close to the border) experimental problems (sometimes, crystals precipitate out of the solution, which give very high signal), or features of your system (a cell can sometimes divide in three instead of two), but they can also be the result of a mechanism that no one's ever considered because it's rare and you're doing research, which means that some of the stuff you do is simply not known yet.

Ideally, you take the time to investigate every outlier, and only remove it from your data set once you understand why it doesn't fit your model. This is time-consuming and subjective in that the reasons are highly dependent on the experiment, but the alternative is worse: If you don't understand where the outliers came from, you have the choice between letting outliers "mess up" your results, or defining some "mathematically rigorous" approach to hide your lack of understanding. In other words, by pursuing "mathematical rigorousness" you choose between not getting a significant effect and not getting into heaven.

EDIT

If all you have is a list of numbers without knowing where they come from, you have no way of telling whether some data point is an outlier, because you can always assume a distribution where all data are inliers.

-
Not all outliers are generated from an experiment, however. I worked with a large dataset that involved the collection of real-estate information in a region (sale price, number of bedrooms, square footage, etc), and every now and then, there would be data entry mistakes and I'd have a 400,000 bedroom house go for 4 dollars, or something nonsensical like that. I would think that part of the goal of determining an outlier is to see whether it's possible to be generated from the data, or if it was just an entry error. –  Christopher Aden Feb 13 '11 at 19:25
@Christopher Aden: I'd consider that part of the experimental process. Basically, in order to be able to remove outliers, you have to understand how the data were generated, i.e. no removing outliers without a good reason. Otherwise you're just stylizing your data. I've edited my answer to reflect this a bit better. –  Jonas Feb 13 '11 at 19:40
This is perfectly reasonable, but assumes you already have a decent amount of prior knowledge about what the true distribution is. I was thinking more in terms of scenarios where you don't and it could be very heavy tailed or bimodal. –  dsimcha Feb 14 '11 at 3:06
@dsimcha: I don't think you can identify outliers in that case (see also my edit). –  Jonas Feb 14 '11 at 4:08
@dsimcha - you always have prior knowledge! for how were the data given to you? you always always know that much. data doesn't magically just show up. and you can always make tentative assumptions. "outliers" based on these assumptions basically give you a clue that something in your assumptions is wrong. by studying the "outlier" (which is always relative) you can improve your model. –  probabilityislogic Feb 16 '11 at 12:50
show 1 more comment

You are correct that removing outliers can look like a subjective exercise but that doesn't mean that it's wrong. The compulsive need to always have a rigorous mathematical reason for every decision regarding your data analysis is often just a thin veil of artificial rigour over what turns out to be a subjective exercise anyway. This is especially true if you want to apply the same mathematical justification to every situation you come across. (If there were bulletproof clear mathematical rules for everything then you wouldn't need a statistician.)

For example, in your long tail distribution situation, there's no guaranteed method to just decide from the numbers whether you've got one underlying distribution of interest with outliers or two underlying distributions of interest with outliers being part of only one of them. Or, heaven forbid, just the actual distribution of data.

The more data you collect, the more you get into the low probability regions of a distribution. If you collect 20 samples it's very unlikely you'l get a value with a z-score of 3.5. If you collect 10,000 samples it's very likely you'll get one and it's a natural part of the distribution. Given the above, how do you decide just because something is extreme to exclude it?

Selecting the best methods in general for analysis is often subjective. Whether it's unreasonably subjective depends on the explanation for the decision and on the outlier.

-
+1 Barnett and Lewis, who wrote the book on outliers, state "an outlier in a set of data [is] an observation (or subset of observations) which appears to be inconsistent with the remainder of that set of data" [at p. 7]. They continue, "It is a matter of subjective judgement on the part of the observer whether or not some observation ... is picked out for scrutiny. ... What characterizes the 'outlier' is its impact on the observer ... ." –  whuber Aug 27 at 21:41
"the book" is slightly ambiguous here. I'd consider Barnett and Lewis the leading monograph, but it's not the only book on outliers. amazon.com/Outlier-Analysis-Charu-C-Aggarwal/dp/1461463955 is recent. There is also an older book by D.M. Hawkins. –  Nick Cox Aug 28 at 8:19

I don't think it is possible to define an outlier without assuming a model of the underlying process giving rise to the data. Without such a model we have no frame of reference to decide whether the data are anomalous or "wrong". The definition of an outlier that I have found useful is that an outlier is an observation (or observations) that cannot be reconciled to a model that otherwise performs well.

-
Hmm...In his EDA text, John Tukey specifically defined outliers without using any models at all. –  whuber Mar 8 '11 at 6:59
You can define outliers without a model, but I have found such definitions to be unhelpful. BTW, by model, I don't necessarily mean a statistical model that has been explicitly fit to the data. Any definition of an outlier requires you to make some assumption about what sort of values you expect to see, and what sort of values you don't expect to see. I think it is better if these assumptions (i.e. the model) are made explicit. There is also the point that in EDA, you are exploring the data, your definition of an outlier may be very different for EDA than for fitting a final model. –  Dikran Marsupial Mar 9 '11 at 11:09

Definition 1: As already mentioned, an outlier in a group of data reflecting the same process (say process A) is an observation (or a set of observations) that is unlikely to be a result of process A.

This definition certainly involves an estimation of the likelihood function of the process A (hence a model) and setting what unlikely means (i.e. deciding where to stop...). This definition is at the root of the answer I gave here. It is more related to ideas of hypothesis testing of significance or goodness of fit.

Definition 2 An outlier is an observation $x$ in a group of observations $G$ such that when modeling the group of observation with a given model the accuracy is higher if $x$ is removed and treated separately (with a mixture, in the spirit of what I mention here).

This definition involves a "given model" and a measure of accuracy. I think this definition is more from the practical side and is more at the origin of outliers. At the Origin, outlier detection was a tool for robust statistics.

Obviously these definitions can be made very similar if you understand that calculating likelihood in the first definition involves modeling and calculation of a score :)

-

There are many excellent answers here. However, I want to point out that two questions are being confused. The first is, 'what is an outlier?', and more specifically to give a "rigorous definition" of such. This is simple:

An outlier is a data point that comes from a different population / distribution / data generating process than the one you intended to study / the rest of your data.

The second question is 'how do I know / detect that a data point is an outlier?' Unfortunately, this is very difficult. However, the answers given here (which really are very good, and which I can't improve upon) will be quite helpful with that task.

-
This is a thought-provoking answer. So, suppose I generate $99$ iid values from a Normal$(0,1)$ distribution--they are likely to span a range from around $-2.5$ to $2.5$--and generate one more value from a Normal$(4,1)$ distribution and it happens to equal $2$ (for which there's about a $1$ in $40$ chance). It's highly unlikely that extra $2$ would be determined to be an outlier. Do you claim that it really is? Your quotation makes me think so, but I don't see how this can be made practically operational. –  whuber Jul 10 '12 at 21:25
@whuber, yes. I say that it is an outlier, although you would never notice it (which, I suspect, is what you mean by practically operational). –  gung Jul 10 '12 at 21:35
I appreciate the distinction you're making. I just wanted to point out the sharp contrast between your definition and most of the other definitions or descriptions of outliers in this thread. Yours does not seem like it could lead to satisfactory practical procedures: you would always have to accept that a huge part of your dataset might be "outlying" but without having any way to detect or resolve that. –  whuber Jul 10 '12 at 21:40
@whuber, I wholeheartedly agree. I see this as loosely analogous to hypothesis testing, where (eg) 2 groups may differ by a very small, undetectable amount, or may differ by a moderate amount, but the samples you ended up with were very similar by chance alone; nonetheless, from a theoretical perspective it's worth understanding & maintaining the distinction. –  gung Jul 11 '12 at 2:20
@whuber, you're right. Some make this distinction, but many aren't clear about these ideas. My position is that there is no meaningful reality of "outlier" other than contaminant. Nonetheless, people should also / instead think about the issue as being concerned about point(s) if your results are driven by them alone (whether they are 'real' or not), & thus your results are very fragile. In short, there is no reason to be concerned about point(s) that are from your population & aren't uniquely driving your results; once you've dealt w/ those 2 issues, there is nothing left to "outlier". –  gung Aug 27 at 22:14

An outlier is a data point that is inconvenient to me, given my current understanding of the process that generates this data.

I believe this definition is as rigorous as can be made.

-
Contrast this to John Tukey's definition (he used the term "outside"): "When we look at some batches of values, we see certain values as apparently straying out far beyond the others. ... It is convenient to have a rule of thumb that picks out certain values as "outside" ..." Later he summarizes this as "...identification of individual values that may be unusual." [EDA, chapter 2]. He emphasizes throughout the book that we are describing data rather than pretending to "understand a process," and that multiple valid descriptions are always possible. –  whuber Aug 27 at 21:22
Similarly, "Outliers are sample values that cause surprise in relation to the majority of the sample" (W.N. Venables and B.D. Ripley. 2002. Modern applied statistics with S. New York: Springer, p.119). However, surprise is in the mind of the beholder and is dependent on some tacit or explicit model of the data. There may be another model under which the outlier is not surprising at all, say, the data really are lognormal or gamma rather than normal. –  Nick Cox Aug 27 at 21:33
@Nick That is consistent with Barnett and Lewis, whom I quote in a comment to John's answer. –  whuber Aug 27 at 21:44
@whuber: You say "Contrast this", which I think means you disagree, but I'm not sure. I'd argue that model-formation -- implicit and naive, perhaps -- is why we see patterns in data, or the man in the moon, or outliers. The model may have no physics/chemistry/economic basis, but we have hypothesized a model. Otherwise, there is no surprise, there is no "outside". –  Wayne Aug 27 at 21:48
Tukey is insistent that in describing data we are not necessarily modeling them. It's fair to extend your definition of "model" to include data description, but then the term becomes almost too general to be useful. From Tukey's point of view (as I interpret it, of course), there is no concern about loss of face nor is there any question of convenience or not. Thus, although I respect your motivation, I think your attitude (as reflected in "face-saving" and "inconvenient") is less constructive than other approaches to this question. –  whuber Aug 27 at 21:54