3
$\begingroup$

I am quite new to data analysis and Machine Learning, that's why I am asking for help for a problem I am facing.

It's an outliers detection problem. I have a quite big amount of data that I need to create a model (meaning that all these data have a standard behaviour, so there are no outliers among them). After this, I would need to check another set of data - clearly of the same type of the first set- and I need to check this data for possible outliers respect to the model identified from the first set.

The data features are categorical and numerical, but I should be able to solve these problem using a OneHotEncoder process.

My idea was to use a Machine Learning (or Neural network) unsupervised method to create clusters for the first set, and check if any data of the 2nd set is an outliers for the clusters. Would that make sense?

Any idea of how I can deal with this problem ?

$\endgroup$
1
$\begingroup$

Traditional unsupervised outlier detection algorithms include principal component analysis, support vector machine and Isolation Forest.

Clustering, for example, k-means, is very related to GMM and hence to autoencoder.

Autoencoder is the core technique for almost all unsupervised anomaly detection algorithms(8.2.3 in the reference paper). I thought we can try to embed the inputs and then add an autoencoder for the embeddings. To tell if the coming input in test time is an outlier we can compare the embedding before the autoencoder and the reconstruction output after the autoencoder by calculating their difference, cosine distance for instance.

We can detect the outliers according to the Interquartile Range Rule(IQR).

Since GMM is a soft-clustering algorithm we can add that atop of the autoencoder as described in this paper: Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection, or just an GMM is enough.

Reference: Deep Learning for Anomaly Detection: A Survey

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I am sorry, I am still confused. What do you mean by "_To tell if the coming input in test time is an outlier we can compare the embedding before the autoencoder and the output after the autoencoder by calculating their difference, cosine distance for instance. _" $\endgroup$ – rob Feb 19 at 13:27
  • 1
    $\begingroup$ We can input the embedding into the autoencoder and try to reconstruct it, and there is a difference or distance between the reconstruction and the input embedding. $\endgroup$ – Lerner Zhang Feb 19 at 14:33
  • $\begingroup$ So you would consider outlier elements that are not well reconstructed? Interesting! $\endgroup$ – Davide ND Feb 19 at 14:34
0
$\begingroup$

Clustering with data that is also categorical is a non trivial problem. The most common algorithms (Kmeans, hierarchical, DBscan) use distances, and using One Hot Encoding is not necessarily a solution as the result distances are somewhat "made-up", and will be biased towards highly cardinal features.

Isolation Forest might be a good option to run directly Outlier Detection. You can train it on your available data, and use it to predict whether new data points that come are outliers.
It is not distance based, but it will still need you to choose an encoding for your categorical variables. The standard option is numerical encoding, but you also might try binary encoding as it should perform better than OHE on this kind of task.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.