0
$\begingroup$

I'm trying out different anomaly detection models and would love to hear opinion on my idea from somebody experienced. My goal is to perform anomaly detection with different models and to give each point in data anomaly score based on the predictions of different models.

A lot of tutorials says that k-means is okay to use for anomaly detection even though it classifies anomalies in clusters. But my thought was to use counter to check the 2-3 clusters that have the least points and to assume that they are anomalous. Could this work out is or is this assumption way too vague?

Another problem is that my data is not labeled so there is no ground truth for me to evaluate the performance of the models. How can I do that or where do I even start?

$\endgroup$

1 Answer 1

1
$\begingroup$

this is my first post ;-)

K-means is not the ideal method for clustering outliers. Seethe following example:

Different data with different clusters

In k-means the number of clusters is fixed (i.e. a hyperparameter of that algorithm). You want outliers to be clustered into a bin which is different from the rest of the data so your choice of k needs to be just right for the number of outliers in your data. In the left and the middle example k is chosen well to capture the outliers nicely. The right example groups one outlire into a larger cluster because it has the lowest distance from any of the clusters.

To overcome this problem you may want to repeat k-means with different values of k and minimize the intra-cluster distance. This is a fair enough approach known as the Ellbow Method. Please see wikipedia for more details on that.

A more elegant solution however is to use a hierarchical clustering algorithm like for example the Agglomerative Clustering. Hierarchical Clustering algorithms are genrally better for clustering outliers and aberrant data. The Agglomerative Clustering starts with every data point being its own cluster and merges clusters which are close enough to each other.

In most implementations you can select both the distance metric (e.g. eucledian) and the distance threshold as hyperparameter. Like for example in the sklearn package in python.

from sklearn.cluster import AgglomerativeClustering
ac = AgglomerativeClustering(distance_threshold=.5)
clusters = ac.fit(data)
print(clusters)

The sklearn implementation offers more hyperparameters to tailor the algorithm to your needs but in my opinion the distance_threshold is the most important for your application.

Hope this helps. Best, Chris

$\endgroup$
4
  • $\begingroup$ hey Chris, great first post! :) yes, I'm aware of Elbow Method to better chose n_clusters for k-means but I didn't know the Agglomerative Clustering model yet so thank you, I'll try it out. though it still doesn't answer my question $\endgroup$
    – Kami
    Sep 29, 2021 at 6:49
  • $\begingroup$ upd: clusters = ac.fit_predict(data) returns me clusters and the fit function only creates AgglomerativeClustering object for me $\endgroup$
    – Kami
    Sep 29, 2021 at 6:57
  • $\begingroup$ ah, I see. You miss an answer to your question regarding the 2-3 smallest clusters. Well, this is important also if you chose to go with different clustering models. You cannot be sure that the smallest clusters are also outliers. Cluster membership is calculated based on a distance metric between clusters. What you are interested in is a distance to the overall average. My recommendation would be to look at the distance of the clusters to the population mean. $\endgroup$
    – Chris
    Sep 29, 2021 at 10:41
  • $\begingroup$ makes sense! thank you @Chris $\endgroup$
    – Kami
    Nov 10, 2021 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.