6
$\begingroup$

From some reading I did online, I understand that there are various methods for determining "similarity" used by different clustering algorithms. I am curious if it is good practice to run multiple clustering algorithms/methods (i.e Hiearchical w/ Ward, single linkage, centroid, etc or maybe even K-means) on a dataset and if there is some automated way to to get a "consensus" of clusters. In other words to get some sense of confidence that the right items are clustered together. Items that tend to cluster together using various methods would be considered valid. For example in my example below G and Z tend to cluster together using multiple methods as do S and F.

Label = what I am clustering; X & Y are my variables I use to cluster; Cluster1-3 are the results of three clustering algorithms.

enter image description here

Edit: I removed a side note I had here regarding how large the actual data set I plan to use might be so as not to detract from the main questions.

Improve this question
$\endgroup$
4
  • 3
    $\begingroup$ Your question is broad, you are not asking anything specific. There are many methods of clustering, many distant measures, many ways to validate clusters. You certainly have to read more about cluster analysis before you feel youself able to make conscious choices among those lots of options. $\endgroup$
    – ttnphns
    Commented Oct 28, 2013 at 17:43
  • $\begingroup$ I agree with ttnphns. If you have so much data, clustering will be messy no matter what algorithm you use. Perhaps try to reduce the dimensions of your dataset (multiple correspondance analysis) $\endgroup$
    – Drew75
    Commented Oct 28, 2013 at 18:58
  • $\begingroup$ Ok. Will let's assume my data set was small enough in terms of # of columns & rows. Say something just a tad larger than the example I included. And say for example I used three or so clustering algorithms and saved the results of each. Obviously each algorithm (and various methods to assess "similiarity") might yield a different set of clusters. But I have seen that some objects tend to be clustered identically or at least similarly by each algorithm. $\endgroup$
    – daniellopez46
    Commented Oct 29, 2013 at 18:23
  • 1
    $\begingroup$ I'm thinking something along the lines of randomForest in terms of the concept of voting. In my example objects G & Z were clustered together by all three algorthms so I would be confident in using this as valid cluster in further analysis. Objects S & F also clustered together. I might then say S&F&R were clustered together by two algorithms so I will decide that that is a valid cluster. My question really is if there something that helps facilitate this process I'm proposing. If you think I am still not being clear but have a suggestion on how I could be clearer please let me know. $\endgroup$
    – daniellopez46
    Commented Oct 29, 2013 at 18:36

1 Answer 1

Reset to default
4
$\begingroup$

As daniellopez46 noted, I think you are thinking of consensus clustering where you basically form an ensemble of different clustering runs. What is a bit strange here is that you would want the ensemble to contain results from different clustering methods which can be very misleading. I say this because unlike supervised learning, unsupervised learning always has in a larger or smaller degree a subjective component as you need to have an idea of what you consider a grouping you would be interested in based on your data. Elaborating a bit, clustering is labeling observations based on what relationship they have with other observations in your feature space. Different clustering algorithms will understand this in a totally different fashion as they are looking for different things. Depending on what kind of topology you are looking for you will (as a human) be satisfied with what one clustering algorithm produced on some data set and be totally dissatisfied with what it did on another data set. Look at this question I recently answered, where you can see a diagram of how different clustering techniques treat the same data sets.

Another thing that should be noted is that consensus clustering is still very new and is basically just being explored so don't take it as panacea.

Improve this answer
$\endgroup$
6
  • $\begingroup$ thanks for the links to the question you previously answered as well as more info on consensus clustering. I think at this point I will play around with consensus clustering for a small data set I have but will also continue my learning using various individual clustering algorithms. I guess I need to learn more about evaluating clustering algorithm output. $\endgroup$
    – daniellopez46
    Commented Nov 12, 2013 at 19:22
  • 1
    $\begingroup$ By the way, if you are familiar with k-means, it handles well large-ish data. Maybe you should explore ensembles of just k-means runs. Another thing, mini-batch k-means updates centers based on random samples of data so it is suitable for very large data and is supposed to sometimes converge to the same optimal points (you can check the paper and implementation in the scikit learn python package). There are a few packages in R for cluster validation associated to good theoretical sources: fpc, clValid. You can also maybe look a bit into separability measures like Jeffries-Matusita. $\endgroup$
    – JEquihua
    Commented Nov 13, 2013 at 15:30
  • $\begingroup$ My actual data set only has three continuous variables and about a dozen or so categorical variables. As far as I know the only method I can use is hierarchical. Do you have a good link to something that explains how one can go about evaluating the validity of clusters produced by hierarchical clustering? $\endgroup$
    – daniellopez46
    Commented Nov 13, 2013 at 17:30
  • $\begingroup$ I'm almost sure clValid has some example for hierarchical clustering. The indices available in fpc for estimating the number of clusters are intended for clustering on continuous type variables. The clusterboot function in the fpc package tries to see how replicable your clustering is on bootstrapped sampes or to data with noise introduced into it, that could be of help to you. It is well documented and points you to this source which is free: Cluster-wise assessment of cluster stability. You could also try methods for mixed-type clustering. $\endgroup$
    – JEquihua
    Commented Nov 13, 2013 at 20:21
  • $\begingroup$ For example by creating principal components, there are some variations for mixed type data or use multiple correspondence analysis on just your categorical variables and then feeding (all) these to k-means. Also random forest in unsupervised mode (labs.genetics.ucla.edu/horvath/RFclustering/RFclustering.htm) could be quite cool. Problem is i'm not sure your data size will allow these kind of things. $\endgroup$
    – JEquihua
    Commented Nov 13, 2013 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.