I'm learning the kmeans to find out anomaly from the dataset. but I don't know how to set threshold. I tried by the putting mean of the centroid to point distance but it's not working, half my record is shown as an anomaly. I know setting threshold depends on what data you have but I want to know , is there any thing from that I find out the threshold or set any value which affects less on my data.
$\begingroup$
$\endgroup$
6
-
$\begingroup$ K-means is sensitive to outliers. I don't think it can be used for anomaly detection. $\endgroup$– Has QUIT--Anony-MousseCommented Aug 11, 2017 at 6:19
-
$\begingroup$ @Anony-Mousse can you suggest me another algorithm for anomaly detection $\endgroup$– NewbieCommented Aug 11, 2017 at 6:37
-
$\begingroup$ Google for outlier detection algorithms. There are plenty: kNN, LOF, Loop,... $\endgroup$– Has QUIT--Anony-MousseCommented Aug 11, 2017 at 6:55
-
$\begingroup$ @Anony-Mousse thnk you i will try $\endgroup$– NewbieCommented Aug 11, 2017 at 6:58
-
$\begingroup$ @Anony-Mousse i want to ask you one thing I'm calculating distance between centroid and point using sqdist is it correct I'm new so in this $\endgroup$– NewbieCommented Aug 11, 2017 at 8:59
|
Show 1 more comment
1 Answer
$\begingroup$
$\endgroup$
Not sure if @Newbie has moved on, but I just saw this post now. I have the same challenge of determining the threshold and the model i had used is explained below. I must put the disclaimer that my approach has not been peer-validated but I am willing to enter into a discussion.
1. Determine the cluster label your dataset closest to. If you have just
one cluster ( n=1), then outliers are the ones farthest from the centroid.
As you rightly said, the challenge is to determine the threshold.
2. approach would be similar to Gaussian model anomaly detection error
analysis where you compare the predicted results with human results and
compute single value error computation using F1-score. You can keep on
refining your threshold until your F1 score is closer to 1 ( > 0.80). This
would have eliminated false positives.
