anomaly detection for clustering data structure

Question

One of anomaly detection algorithms is to use multivariate Gaussian to construct a probability density, according to Andrew Ng's coursera lecture.

I previously thought that, the probability density method is supervised learning since we build a probability density and determine the threshold (if smaller, then an outlier) based on the positive samples. But somehow I read somewhere, that anomaly detection could be unsupervised learning by only training negative sample. Or I am totally wrong?

What if data show clustering structures (not a single chunk)? In this case do we resort to unsupervised clustering to construct the density? If yes, how to do it? Are there other systematic ways to discover if such a case exists?

Answer 1

As answered before (don't cross-post duplicates, please!)

You can just use regular GMM and use a manualky chosen threshold on the likelihood to identify outliers. Points that don't fit the model well are outliers. The threshold can be interpreted as a probability, e.g. 99.7% of points should be less than 3.

This works okay as long as your data really is composed of Gaussians.

Furthermore, clustering is fairly expensive. Usually it will be faster to directly use a nonparametric outlier detrctor such as KNN or LOF or LOOP. These are unsupervised.

There are also methods such as one-class SVMs that are supposed to be trained on known outlier free data, i.e., one-class only. If you train such models on data with outliers, it may learn these instances as "normal".

Andrew Ng's Coursera lecture on this extremely one-sided, and ignores most of the work. You need to look at some surveys of outlier detection to get a wider non-ML view.

https://scholar.google.com/scholar?hl=en&q=unsupervised+outlier+detection+survey

Answer 2

Anomaly detection can be supervised, unsupervised, or semi-supervised. In the supervised case, each point in the training set has a given label that says whether or not it's an anomaly. The anomaly detector is trained to correctly reproduce these labels. In the unsupervised case, no labels are available, and anomalies are defined as points that are dissimilar in some sense to most of the points in the training set. In the semi-supervised case, a subset of training points is labeled, and the anomaly detector is trained to correctly identify the labeled points, but also take advantage of information in the unlabeled points.

What if data show clustering structures (not a single chunk)? In this case do we resort to unsupervised clustering to construct the density? If yes, how to do it?

If labels are available, it would make sense to take advantage of them by training in the supervised or semi-supervised regime. If there are no labels, there's no choice but to go unsupervised. In either case, one could explicitly try to account for clustered structure or not. For example, methods like Gaussian mixture models treat clusters explicitly. Alternatively, methods like one-class SVMs and isolation forests can handle clustered data, but don't explicitly model clusters.

Answer 3

1) Super-vised or unsupervised with examples- https://machinelearningstories.blogspot.com/2018/07/anomaly-detection-anomaly-detection-by.html It can be any one of 3 as suggested by @user20160

2 ) Define abnormality based on your application/process. Simple quantile distribution may give you better results compare to PCA based anomaly detection.

3) K-means, LOF, ABOD( angle based) , CBOD( connectivity based), PCA based, NN , AutoEncoder based anomaly detection/Outlier detection techniques are present. { Broadly categorised as distance based, density based and relation based)