AFAIK - One way to process data faster and more efficiently is to detect abnormal events, changes, or shifts in datasets. Anomaly detection, also known as outlier detection is the process of identifying extreme points or observations that are significantly deviating from the remaining data. By analyzing the extreme points one can understand the extreme working conditions of the system.

I am really having trouble understanding

Anomaly detection is Supervised or Un-supervised and What is the best technique used for Anomaly Detection?

  • $\begingroup$ It is almost always a mistake to remove 'outliers' unless you have knowledge separately from the data that a value results from a documentable error (data entry, machine failure) or is obviously impossible (negative height, human age over 175). For data analysis there are robust measures that (temporarily) mitigate the effect of unusual observations (e.g, trimmed means for location, MAD for variation.) $\endgroup$
    – BruceET
    Jun 29 '20 at 17:05
  • $\begingroup$ @BruceET: The OP is not about removing outliers. $\endgroup$
    – Michael M
    Jun 29 '20 at 17:50

Typically, it is unsupervised. But actually it can be either. Let's start with supervised anomaly detection.

Supervised anomaly/outlier detection

For supervised anomaly detection, you need labelled training data where for each row you know if it is an outlier/anomaly or not. Any modeling technique for binary responses will work here, e.g. logistic regression or gradient boosting.

The typical application is fraud detection.

Usually, one does not have labelled data, so one has to rely on unsupervised methods with their usual pros and cons.

Unsupervised anomaly/outlier detection

We have a "reference" training data at hand but unfortunately without knowing which rows are outliers or not. Here, it is tempting to let statistical algorithms do the guess work. Some of the typical approaches are:

  • density based: local outlier factor (LOF), isolation forests.

  • distance based: How far away is a row from the average e.g in terms of Mahalanobis distance?

  • autoencoder: How bad can the row be reconstructed by an autoencoder neural network?

  • model based: model each variable by the others and hunt for high residuals.

  • ...

Each of the techniques has its pros and cons. There is no approach that does somehow better than the rest for all types of problems.

Note about dimensions and unsupervised detection algos

For 1-2 dimensional data, you can plot the data and visually identify outliers/anomalies as points far away from the rest. For very high dimensional data, unsupervised anomaly detection is close to being a hopeless task due to the curse of dimensionality, which - in the sense of anomaly detection - means that every point eventually becomes an outlier.

  • $\begingroup$ Once we introduce labeled data how would outlier detection be distinct from classification? $\endgroup$
    – Ryan Volpi
    Jun 29 '20 at 18:56
  • 1
    $\begingroup$ It becomes an application of classification, see the part on supervised anomaly detection. $\endgroup$
    – Michael M
    Jun 29 '20 at 19:05
  • 1
    $\begingroup$ To add, I found Wikipedia spelled out the distinction of supervised outlier detection succinctly: "the key difference to many other statistical classification problems is the inherent unbalanced nature of outlier detection." Though, perhaps that is an obvious point. $\endgroup$
    – Ryan Volpi
    Jun 29 '20 at 19:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.