1
$\begingroup$

I have a table of similarities (cosines) and I clustered it with the Ward method. Great outcomes, a wonderful dendogram, but then I tried to evaluate the quality of this cluster solution and I got stuck.

First: identifying the number of clusters in my data (cause in Ward is not like k-means where you have to set a precise number of clusters). I calculated the sum of squares (see attachment) to see how many clusters are there, but there isn't a proper "elbow" in the data, so how many clusters shall I consider?

Second: trying to calculate the purity of the clustering (with the tool CluTo), by indicating 4, 5, 6, 7... clusters, I can see that the purity increases the more clusters I indicate. Of course. If the number of clusters equals the number of instances of my data, then purity is 1 (the maximum). dah.

Any suggestion on how to report this? (number of clusters? quality of the clustering solution?)

enter image description here

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ What software are you using? Most software offers some additional statistics that can help make this decision (e.g. the Cubic Clustering Criterion - CCC). $\endgroup$
    – Peter Flom
    Commented Jul 16, 2014 at 11:45
  • $\begingroup$ Great outcomes, a wonderful dendogram Ward's method always gives pleasant dendrograms, even if there is hardly any clusters in the data. $\endgroup$
    – ttnphns
    Commented Jul 16, 2014 at 17:15
  • 1
    $\begingroup$ It is better to plot some sort of ratio between the within-sum-of-squares and the between-sum-of-squares. That makes up several clustering validity criterions, such as Calinski-Harabasz, Davies-Bouldin, Gap index, CCC (mentioned by Peter). You may search this site and the web to read about them. $\endgroup$
    – ttnphns
    Commented Jul 16, 2014 at 17:20

1 Answer 1

Reset to default
1
$\begingroup$

If you are using R, this is a really nice page (http://www.statmethods.net/advstats/cluster.html) that steps though a few different methods to help in identifying the optimal number of clusters. HTH.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.