What is the difference between Outlier and Anomaly in the context of machine learning. My understanding is that both of them refer to the same thing.
3$\begingroup$ Out of curiosity, where in the literature is such a distinction made? I was under the impression that "outliers" have no formal definition, outside of being high leverage and high influence observations. Influence and leverage do have mathematical definitions, but considering something "high" is arbitrary. It seems like arbitrary words are being swapped around. $\endgroup$– AdamOJan 7, 2016 at 18:26
$\begingroup$ People who use the word "inlier" implicitly make some kind of distinction between "anomaly" and "outlier," because an in inlier is a kind of anomaly. Since neither "outlier" nor "anomaly" have definite, commonly understood technical definitions, we should expect this question to have multiple answers that differ (at least slightly) from each other. $\endgroup$– whuber ♦Nov 15, 2017 at 19:29
The two terms are synonyms according to:
Aggarwal, Charu C. Outlier Analysis. Springer New York, 2017, doi: http://dx.doi.org/10.1007/978-3-319-47578-3_1
Quotation from page 1:
Outliers are also referred to as abnormalities, discordants, deviants, or anomalies in the data mining and statistics literature.
Bold text is not part of the original text.
The free to download pdf of the book available from the author is here.
$\begingroup$ The fact that "outliers" are referred to as "anomalies" does not mean that they are synonymous. "Dogs" are sometimes referred to as "animals", for that matter. I tried to address this in more detail in this answer (I couldn't post it here, because the question is protected). $\endgroup$– Marco13Aug 12, 2018 at 15:08
A tongue-in-cheek answer:
Outlier: a value that you predictably find in your data that indicates your model does not work properly
Anomaly: a value that against all odds you find in your data that indicates your model does work properly
A more serious, less cryptic answer:
The concept of outliers starts from the issue of building a model that makes assumptions about the data. Outliers are often indicators that the model does not describe the data properly and thus we should question the results of our model or quality of our data.
The concept of anomalies starts outside the theoretic world and inside the applied world: we want to look for unusual behavior in our data, sometimes motivated by the fact that we are interested in finding behavior that someone is trying to hide (like a virus in an email). The problem is that since people are trying to hide what they are doing, we don't really know what to look for. So we take a set of "good" data, and decide that whatever we find in our new dataset that doesn't look "good" is an anomaly and worth our time to checkout in more detail. Often, looking for anomalies means looking for outliers in your new data set. But note that these values may be very common in your new dataset, despite being rare in your old dataset!
In summary, the two concepts are very similar in terms of the statistics behind them (i.e. unusual values given your fitted model) but come at the idea from different angles. In addition, when we talk about outliers, we typically mean an unusual data point in the data used to fit our model, where as an anomaly is usually meant as an unusual data point in a dataset outside of the data used to fit our model.
Note: this answer is based on how I've seen the two terms frequently used rather than formal definitions. User experiences may differ.
An anomaly is a result that can't be explained given the base distribution (an impossibility if our assumptions are correct). An outlier is an unlikely event given the base distribution (an improbability).
7$\begingroup$ Quoting source for the definitions and example would highly improve the answer. $\endgroup$– Tim ♦Jan 7, 2016 at 8:54
4$\begingroup$ As far as I know they are synonyms. So @H. Iqbal really must quote the source and all readers must then evaluate the authoritativeness of sayd source $\endgroup$ Jan 7, 2016 at 11:48
2$\begingroup$ Impossibility seems to imply P(X = ANOMALY) = 0 (i.e exactly 0). My understanding of anomaly detection is that the researcher may be interested in events that may have positive probability. $\endgroup$– Cliff ABJan 7, 2016 at 17:42
The terms are largely used in an interchangeable way. "Outlier" refers to something lying outside the norm - so it is "anomalous". But I have the inpression that "outlier" is usually used for very rare observations. In statistics, on a normal distribution, you would consider three sigma to be outliers. That is 99.7% of your objects are expected to be "normal". "Anomaly" is used much more liberally. If you suddenly have millions of visitors on your website, these are not rare visitors. The sudden increase in visitors however is still "anomalous", whereas each individual visitor is not an "outlier".
It may have been in this article where I saw these differences discussed, but I can't access it right now, unfortunately.
Statistical Analysis and Data Mining, Volume 5, Issue 5, October 2012, Pages 363–387 A survey on unsupervised outlier detection in high-dimensional numerical data
1$\begingroup$ I think you've subtly hinted at the difference between outliers and anomalies; outliers are used to describe data that doesn't fit a general trend, anomalies describe unusual traffic on a server. 50% jk. $\endgroup$– Cliff ABJan 7, 2016 at 13:54
Just to muddy the waters further, in climatology anomaly just implies the difference between value and mean, or a deviation:
The term temperature anomaly means a departure from a reference value or long-term average. A positive anomaly indicates that the observed temperature was warmer than the reference value, while a negative anomaly indicates that the observed temperature was cooler than the reference value.
That may well be regarded as outside machine learning, but people interested in the question may be interested in this.
Good question. However, google search on "difference between outliers and anomalies site:.edu" shows that there is no theoretical difference between these two terms. They are being used interchangeably in literature.
An outlier is a data point that makes it hard to fit a model. You face outliers, often unwillingly, when you are trying to fit a model on your dataset. Removing outliers enables building better (i.e. more generalizable) models. A point $(1,5)$ would be an outlier for the model $y=x$. You ignore it in light of the fact that all your other points $(1,1)$, $(5,5)$, $(3,3.1)$ more closely fit $y=x$.
An anomaly can be one data point, or also a general trend or behavior observed in data after a model has already been built or an understanding of the data-generating process formed. You face anomalies because the system starts behaving differently, or you seek out such data points, because you want to be informed when an event occurs during which your model is not valid. You may care about observing any anomalous behavior in amplitudes of ocean waves, not because you want to throw away those data points and build a better model, but because you want to be aware when a tsunami might be happening.
2$\begingroup$ I disagree with most of this. First, the first sentence can be your definition of outlier if you like, but it's hard to reconcile with many other definitions or usages. If the data are (1, 1), (2, 2), (3, 3), (much bigger, much bigger) then the much bigger point would often be described as an outlier but there is no problem fitting a model. You might (and should) wonder why the data come that way, but fitting a model is easy. More generally, the principle is that an outlier may be separated from the main body of the data but still consistent with a plausible model. $\endgroup$ May 4, 2017 at 10:42
$\begingroup$ Second, if the implication that omitting outliers is just what you should do, then (a) it is often problematic even to say which the outliers are (b) there are many other solutions. The thread stats.stackexchange.com/questions/78063/… ranges more widely than its title to mention several. $\endgroup$ May 4, 2017 at 10:45
1$\begingroup$ If you follow my link, you'll see that I've already posted at some length on outliers. I don't get any sense on re-reading your answer that you are thinking retrospectively as you seem to be talking about removing outliers while fitting. On re-reading, I note also that the first sentence of your second paragraph includes the idea that an anomaly can be 'a general trend or behaviour', which is unlikely to be what you mean -- or if it is, I don't understand it. $\endgroup$ May 4, 2017 at 12:31