1
$\begingroup$

I'm learning the kmeans to find out anomaly from the dataset. but I don't know how to set threshold. I tried by the putting mean of the centroid to point distance but it's not working, half my record is shown as an anomaly. I know setting threshold depends on what data you have but I want to know , is there any thing from that I find out the threshold or set any value which affects less on my data.

$\endgroup$
6
  • $\begingroup$ K-means is sensitive to outliers. I don't think it can be used for anomaly detection. $\endgroup$ Aug 11, 2017 at 6:19
  • $\begingroup$ @Anony-Mousse can you suggest me another algorithm for anomaly detection $\endgroup$
    – Newbie
    Aug 11, 2017 at 6:37
  • $\begingroup$ Google for outlier detection algorithms. There are plenty: kNN, LOF, Loop,... $\endgroup$ Aug 11, 2017 at 6:55
  • $\begingroup$ @Anony-Mousse thnk you i will try $\endgroup$
    – Newbie
    Aug 11, 2017 at 6:58
  • $\begingroup$ @Anony-Mousse i want to ask you one thing I'm calculating distance between centroid and point using sqdist is it correct I'm new so in this $\endgroup$
    – Newbie
    Aug 11, 2017 at 8:59

1 Answer 1

3
$\begingroup$

Not sure if @Newbie has moved on, but I just saw this post now. I have the same challenge of determining the threshold and the model i had used is explained below. I must put the disclaimer that my approach has not been peer-validated but I am willing to enter into a discussion.

 1. Determine the  cluster label your dataset closest to. If you have just 
 one cluster ( n=1), then outliers are the ones farthest from the centroid. 
 As you rightly said, the challenge is to determine the threshold.

 2. approach would be similar to Gaussian model anomaly detection error 
 analysis where you compare the predicted results with human results and 
 compute single value error computation using F1-score. You can keep on 
 refining your threshold until your F1 score is closer to 1 ( > 0.80). This 
 would have eliminated false positives. 
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.