4
$\begingroup$

I’m working on a task regarding cluster analysis for about half a year now, but since the fields of pattern recognition and cluster analysis are quite complex ones, I would call myself a beginner in this subject.

I’m trying to cluster some production data consisting of time series and scalar values. For the feature extraction I’ve used expert knowledge for identifying significant points in the time series and the meaningful scalar values. This lead to a feature space of approximately 25 to 30 dimensions, with all features normalized into a consistent dynamic range.

So far I used to work with labeled data sets, knowing the clustering structure of the data. I’ve evaluated different clustering algorithms using this data set. For example k-means and k-medoids lead to sensible clustering results, representing the expected structure. In most cases the Gaussian mixture models performed more or less poorly and the DBSCAN completely failed (especially since it is hard to find decent parameters for the DBSCAN). I’ve also tested some fuzzy algorithms like the Gath-Gave and the Gustafson-Kessel, which performed quite well too.

Now I’ve read this How to understand the drawbacks of K-means post, which I really liked, cause it’s quite instructive and highly interesting to read! So my concerns now are: I’ve found for example the k-medoids algorithm greatly performant and gained good clustering results for my labeled data sets. But from now on I’ll work on unlabeled data sets, which I know almost nothing about. For example I don’t know the cluster shapes, cause I don’t see a way to guess the shapes in a 30 dimensional feature space. So how do I know if the expected spherical cluster shapes of a k-medoids fit my data. By the way, I didn’t know the cluster shapes for my labeled data sets either. Furthermore I don’t know the cluster volumes expected. Maybe some single clusters have two to ten time of the objects of the remaining clusters. Where k-medoids etc. expect clusters of the same volume (At least if I got that right?).

In the mentioned post it says “Too easy to use badly: All in all, it's too easy to throw k-means on your data, and nevertheless get a result out (that is pretty much random, but you won't notice). I think it would be better to have a metod which can fail if you haven't understood your data...”. So the main points of my question are: How do I know which Algorithm to use? When do I get decent clustering results, or even not random ones? And all that without really knowing the structure of the data set!

I’m really not expecting comprehensive answers to my issues, because I know that I can expect no or at least only hardly objective results using cluster analysis. But I hope to find some good hints, helping me to think about different directions to face the issues of my clustering task!

Thanks in advance for your answers!

$\endgroup$
1
$\begingroup$

Make sure to visualize your data. Scatter plot grids can be helpful, but also try dimensionality reduction with PCA and MDS.

Try to find an explanation for your clusters. For example, you could train a decision tree on your clusters. If the tree has a reasonable accuracy, you can try to analyze it to figure out which features have the largest impact on your clusters. This may be helpful to eventually identify classes you want to have, and prune clusters that are not meaningful or useful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.