# Rigorous definition of an outlier?

People often talk about dealing with outliers in statistics. The thing that bothers me about this is that, as far as I can tell, the definition of an outlier is completely subjective. For example, if the true distribution of some random variable is very heavy-tailed or bimodal, any standard visualization or summary statistic for detecting outliers will incorrectly remove parts of the distribution you want to sample from. What is a rigorous definition of an outlier, if one exists, and how can outliers be dealt with without introducing unreasonable amounts of subjectivity into an analysis?

-
If you want to know for a specific distribution then ask about your example. It will be different for different situations. – John Feb 13 '11 at 15:55
Well, I would expect that you will have a rigorous definition of an outlier when you'll be able to define unreasonable amounts of subjectivity objective manner ;-), Thanks – eat Feb 13 '11 at 16:00
But the definition can vary by underlying distribution and situation. I could say ± 1.5 IQR, or 3 SD, or some such. But I could take a totally different approach if I have two kinds of measures, say reaction time and accuracy. I can say RT's conditioned on a level of accuracy. They can all be good and mathematically rigorous and have different applications and meanings. – John Feb 13 '11 at 16:08
There are MANY rigorous definitions of outlier. But the choice among those can seem arbitrary. But I think this is part of the misconception that statistics is a subject in which each problem has one correct answer. – Peter Flom Mar 8 '11 at 1:55

As long as your data comes from a know distribution with known properties, you can rigorously define an outlier as an event that is too unlikely to have been generated by the observed process (if you consider "too unlikely" to be non-rigorous, then all hypothesis testing is).

However, this approach is problematic on two levels: It assumes that the data comes from a known distribution with known properties, and it brings the risk that outliers are looked at as data points that were smuggled into your data set by some magical faeries.

In the absence of magical data faeries, all data comes from your experiment, and thus it is actually not possible to have outliers, just weird results. These can come from recording errors (e.g. a 400000 bedroom house for 4 dollars), systematic measurement issues (the image analysis algorithm reports huge areas if the object is too close to the border) experimental problems (sometimes, crystals precipitate out of the solution, which give very high signal), or features of your system (a cell can sometimes divide in three instead of two), but they can also be the result of a mechanism that no one's ever considered because it's rare and you're doing research, which means that some of the stuff you do is simply not known yet.

Ideally, you take the time to investigate every outlier, and only remove it from your data set once you understand why it doesn't fit your model. This is time-consuming and subjective in that the reasons are highly dependent on the experiment, but the alternative is worse: If you don't understand where the outliers came from, you have the choice between letting outliers "mess up" your results, or defining some "mathematically rigorous" approach to hide your lack of understanding. In other words, by pursuing "mathematical rigorousness" you choose between not getting a significant effect and not getting into heaven.

EDIT

If all you have is a list of numbers without knowing where they come from, you have no way of telling whether some data point is an outlier, because you can always assume a distribution where all data are inliers.

-
Not all outliers are generated from an experiment, however. I worked with a large dataset that involved the collection of real-estate information in a region (sale price, number of bedrooms, square footage, etc), and every now and then, there would be data entry mistakes and I'd have a 400,000 bedroom house go for 4 dollars, or something nonsensical like that. I would think that part of the goal of determining an outlier is to see whether it's possible to be generated from the data, or if it was just an entry error. – Christopher Aden Feb 13 '11 at 19:25
@Christopher Aden: I'd consider that part of the experimental process. Basically, in order to be able to remove outliers, you have to understand how the data were generated, i.e. no removing outliers without a good reason. Otherwise you're just stylizing your data. I've edited my answer to reflect this a bit better. – Jonas Feb 13 '11 at 19:40
This is perfectly reasonable, but assumes you already have a decent amount of prior knowledge about what the true distribution is. I was thinking more in terms of scenarios where you don't and it could be very heavy tailed or bimodal. – dsimcha Feb 14 '11 at 3:06
@dsimcha: I don't think you can identify outliers in that case (see also my edit). – Jonas Feb 14 '11 at 4:08
@dsimcha - you always have prior knowledge! for how were the data given to you? you always always know that much. data doesn't magically just show up. and you can always make tentative assumptions. "outliers" based on these assumptions basically give you a clue that something in your assumptions is wrong. by studying the "outlier" (which is always relative) you can improve your model. – probabilityislogic Feb 16 '11 at 12:50
show 1 more comment

You are correct that removing outliers can look like a subjective exercise but that doesn't mean that it's wrong. The compulsive need to always have a rigorous mathematical reason for every decision regarding your data analysis is often just a thin veil of artificial rigour over what turns out to be a subjective exercise anyway. This is especially true if you want to apply the same mathematical justification to every situation you come across. (If there were bulletproof clear mathematical rules for everything then you wouldn't need a statistician.)

For example, in your long tail distribution situation, there's no guaranteed method to just decide from the numbers whether you've got one underlying distribution of interest with outliers or two underlying distributions of interest with outliers being part of only one of them. Or, heaven forbid, just the actual distribution of data.

The more data you collect, the more you get into the low probability regions of a distribution. If you collect 20 samples it's very unlikely you'l get a value with a z-score of 3.5. If you collect 10,000 samples it's very likely you'll get one and it's a natural part of the distribution. Given the above, how do you decide just because something is extreme to exclude it?

Selecting the best methods in general for analysis is often subjective. Whether it's unreasonably subjective depends on the explanation for the decision and on the outlier.

-

Definition 1: As already mentioned, an outlier in a group of data reflecting the same process (say process A) is an observation (or a set of observations) that is unlikely to be a result of process A.

This definition certainly involves an estimation of the likelyhood function of the process A (hence a model) and setting what unlikely means (i.e. deciding where to stop...). This definition is at the root of the answer I gave here. It is more related to ideas of hypothesis testing of significance or goodness of fit.

Definition 2 An outlier is an observation $x$ in a group of observations $G$ such that when modeling the group of observation with a given model the accuracy is higher if $x$ is removed and treated separately (with a mixture, in the spirit of what I mention here)

This definition involves a "given model" and a measure of accuracy. I think this definition is more from the practical side and is more at the origin of outliers. At the Origin, outliers detection was a tool for robust statistic.

Obviously these definition can be made very similar if you understand that calculating likelihood in the first definition involves modeling and calculation of a score :)

-

I don't think it is possible to define an outlier without assuming a model of the underlying process giving rise to the data. Without such a model we have no frame of reference to decide whether the data are anomalous or "wrong". The definition of an outlier that I have found useful is that an outlier is an observation (or observations) that cannot be reconciled to a model that otherwise performs well.

-
Hmm...In his EDA text, John Tukey specifically defined outliers without using any models at all. – whuber Mar 8 '11 at 6:59
You can define outliers without a model, but I have found such definitions to be unhelpful. BTW, by model, I don't necessarily mean a statistical model that has been explicitly fit to the data. Any definition of an outlier requires you to make some assumption about what sort of values you expect to see, and what sort of values you don't expect to see. I think it is better if these assumptions (i.e. the model) are made explicit. There is also the point that in EDA, you are exploring the data, your definition of an outlier may be very different for EDA than for fitting a final model. – Dikran Marsupial Mar 9 '11 at 11:09

There are many excellent answers here. However, I want to point out that two questions are being confused. The first is, 'what is an outlier?', and more specifically to give a "rigorous definition" of such. This is simple:

An outlier is a data point that comes from a different population / distribution / data generating process than the one you intended to study / the rest of your data.

The second question is 'how do I know / detect that a data point is an outlier?' Unfortunately, this is very difficult. However, the answers given here (which really are very good, and which I can't improve upon) will be quite helpful with that task.

-
 This is a thought-provoking answer. So, suppose I generate $99$ iid values from a Normal$(0,1)$ distribution--they are likely to span a range from around $-2.5$ to $2.5$--and generate one more value from a Normal$(4,1)$ distribution and it happens to equal $2$ (for which there's about a $1$ in $40$ chance). It's highly unlikely that extra $2$ would be determined to be an outlier. Do you claim that it really is? Your quotation makes me think so, but I don't see how this can be made practically operational. – whuber♦ Jul 10 '12 at 21:25 @whuber, yes. I say that it is an outlier, although you would never notice it (which, I suspect, is what you mean by practically operational). – gung Jul 10 '12 at 21:35 I appreciate the distinction you're making. I just wanted to point out the sharp contrast between your definition and most of the other definitions or descriptions of outliers in this thread. Yours does not seem like it could lead to satisfactory practical procedures: you would always have to accept that a huge part of your dataset might be "outlying" but without having any way to detect or resolve that. – whuber♦ Jul 10 '12 at 21:40 @whuber, I wholeheartedly agree. I see this as loosely analogous to hypothesis testing, where (eg) 2 groups may differ by a very small, undetectable amount, or may differ by a moderate amount, but the samples you ended up with were very similar by chance alone; nonetheless, from a theoretical perspective it's worth understanding & maintaining the distinction. – gung Jul 11 '12 at 2:20 I also think that this leads us to a way of re-thinking the issue. Maybe what we need to worry about is not so much outliers per se, but high-influence data points, as they have the power to distort betas & flip decisions. (Again, in practice, the intersection of true outliers & high-influence points is likely to be close to the union of the sets, but I think the distinction is worth maintaining for the conceptual clarity it affords.) – gung Jul 11 '12 at 2:20

define an outlier as a member of that minimal set of elements which must be removed from a datasetof size n in order to assure 100% compliance with RUM tests conducted at 95% confidence level on all (2^n -1) unique subsets of the data. See Karian and Dudewicz text on fitting data to pdfs using R(Sept 2010) for definition of the RUM test.

-

Outliers are important only in the frequentist realm. If a single datapoint adds bias to your model which is defined by an underlying distribution predeterimined by your theory, then it is an outlier for that model. The subjectivity lies in the fact that if your theory posits a different model, then you can have a different set of points as outliers.

-
 Are you claiming outliers are unimportant in Bayesian data analysis? – whuber♦ Mar 8 '11 at 7:00