Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I understand that fuzzy clustering using FCM produces a membership matrix for the set of data points we feed to it. What characteristics will an anomalous cluster produced during this method have? (Considering I only have unlabelled data)

share|improve this question

migrated from security.stackexchange.com Jul 29 '13 at 8:02

This question came from our site for Information security professionals.

2 Answers 2

I don't think FCM is a particularly suitable clustering method for detection of small numbers of anomalous points: the problem (inherited from k-means clustering) is that it is not well suited for problems where you expect tiny clusters among large clusters. Unless you hit a point of the tiny cluster with your start centroids, you may not detect this tiny cluster even if it is far away from the other points.

If you think the anomalous points are few, but can reasonably be expected to be far away from the majority in some way, then I'd recommend hierarchical cluster analysis which will not miss these points. You may want to reduce the number of data points beforehand by somehow aggregating known "normal" points.

Here's an example in literature (totally different subject) where a small cluster is overlooked (unless lucky initial values are chosen) by k-means and FCM: Bonifacio, et al.: Chemical imaging of articular cartilage sections with Raman mapping, employing uni- and multi-variate methods for data analysis. The Analyst, 2010, 135, 3193-3204, DOI: 10.1039/c0an00459f

It will be very hard to detect few points that are not even far away from the majority unless you put further external information (= knowledge about the problem) into your detection method.

I'm not sure whether a closed-world setup (memberships sum to 1) is appropriate here: if it were supervised, one would speak of one-class problems: a defined "normal" group vs. anomalous/suspicious/out of specification points. This would even be considered a classification (supervised) problem, if you have reference labels only for the normal class. The distinction anomalous vs. OK is an ill-defined problem for a closed world model.

share|improve this answer

No quite sure what you mean by an anomalous cluster.

Do you mean a cluster which contains only outliers? Outliers are points which lie far from the center of the data, for example the top or bottom percentile of your dataset. In general you would expect that a cluster formed of outliers would contain few observations and lie far from the center of the data.

Also to better answer your question can you tell me if the rows of the membership matrix add up to 1?

share|improve this answer
    
I am specifically looking at security anomalies on my dataset, so it might be outliers or the points might be tending to look like other "normal" events. Also, yes, the rows of the membership matrix add up to 1. –  DaTaBomB Jul 31 '13 at 15:47
    
Could you maybe provide me with some example data to illustrate what you consider to be an anomaly? Do you have training data? –  pontikos Aug 5 '13 at 10:38
    
Also this is not really related to anomaly detection but since your membership matrix contains probabilities which sum to 1, one criteria you might consider to assess the quality of your clustering, is what proportion of points have membership probability greater than some threshold (say 95%). –  pontikos Aug 8 '13 at 7:48
    
I'm dealing with network data. Unfortunately I dont have any examples to share with you, but it often happens that there is a "normal" cluster which overlaps with an "anomalous" cluster. I'm interested to know which cluster is normal and which isn't (I have unlabeled data) and what the datapoints in the overlap region mean. –  DaTaBomB Aug 12 '13 at 5:20
    
Well if you look at the membership values that tells to which cluster an element is more likely to belong too. Assuming you know whether a cluster is "anomalous" or "normal" then you could use a 95% cutoff value to classify an element. –  pontikos Aug 12 '13 at 22:52

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.