Outlier detection after clustering

Question

I am quite new to data analysis and Machine Learning, that's why I am asking for help with a problem I am facing.

It's an outliers detection problem. I have a quite big amount of data that I need to create a model (meaning that all these data have a standard behaviour, so there are no outliers among them). After this, I would need to check another set of data - clearly of the same type of the first set- and I need to check this data for possible outliers with respect to the model identified from the first set.

The data features are categorical and numerical, but I should be able to solve this problem using an OneHotEncoder process.

My idea was to use a Machine Learning (or Neural network) unsupervised method to create clusters for the first set, and check if any data of the 2nd set is an outlier for the clusters. Would that make sense?

Any idea of how I can deal with this problem?

Answer 1

Traditional unsupervised outlier detection algorithms include principal component analysis, support vector machine and Isolation Forest.

Clustering, for example, k-means, is very related to GMM and hence to autoencoder.

Autoencoder is the core technique for almost all unsupervised anomaly detection algorithms(8.2.3 in the reference paper). I thought we can try to embed the inputs and then add an autoencoder for the embeddings. To tell if the coming input in test time is an outlier we can compare the embedding before the autoencoder and the reconstruction output after the autoencoder by calculating their difference, cosine distance for instance.

We can detect the outliers according to the Interquartile Range Rule(IQR).

Since GMM is a soft-clustering algorithm we can add that atop of the autoencoder as described in this paper: Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection, or just an GMM is enough.

Reference: Deep Learning for Anomaly Detection: A Survey

Answer 2

Clustering with data that is also categorical is a non trivial problem. The most common algorithms (Kmeans, hierarchical, DBscan) use distances, and using One Hot Encoding is not necessarily a solution as the result distances are somewhat "made-up", and will be biased towards highly cardinal features.

Isolation Forest might be a good option to run directly Outlier Detection. You can train it on your available data, and use it to predict whether new data points that come are outliers.
It is not distance based, but it will still need you to choose an encoding for your categorical variables. The standard option is numerical encoding, but you also might try binary encoding as it should perform better than OHE on this kind of task.