I understand that fuzzy clustering using FCM produces a membership matrix for the set of data points we feed to it. What characteristics will an anomalous cluster produced during this method have? (Considering I only have unlabelled data)
migrated from security.stackexchange.com Jul 29 '13 at 8:02
This question came from our site for information security professionals.
I don't think FCM is a particularly suitable clustering method for detection of small numbers of anomalous points: the problem (inherited from k-means clustering) is that it is not well suited for problems where you expect tiny clusters among large clusters. Unless you hit a point of the tiny cluster with your start centroids, you may not detect this tiny cluster even if it is far away from the other points.
If you think the anomalous points are few, but can reasonably be expected to be far away from the majority in some way, then I'd recommend hierarchical cluster analysis which will not miss these points. You may want to reduce the number of data points beforehand by somehow aggregating known "normal" points.
Here's an example in literature (totally different subject) where a small cluster is overlooked (unless lucky initial values are chosen) by k-means and FCM: Bonifacio, et al.: Chemical imaging of articular cartilage sections with Raman mapping, employing uni- and multi-variate methods for data analysis. The Analyst, 2010, 135, 3193-3204, DOI: 10.1039/c0an00459f
It will be very hard to detect few points that are not even far away from the majority unless you put further external information (= knowledge about the problem) into your detection method.
I'm not sure whether a closed-world setup (memberships sum to 1) is appropriate here: if it were supervised, one would speak of one-class problems: a defined "normal" group vs. anomalous/suspicious/out of specification points. This would even be considered a classification (supervised) problem, if you have reference labels only for the normal class. The distinction anomalous vs. OK is an ill-defined problem for a closed world model.
No quite sure what you mean by an anomalous cluster.
Do you mean a cluster which contains only outliers? Outliers are points which lie far from the center of the data, for example the top or bottom percentile of your dataset. In general you would expect that a cluster formed of outliers would contain few observations and lie far from the center of the data.
Also to better answer your question can you tell me if the rows of the membership matrix add up to 1?