Hierarchical clustering and Dendrogram interpretation

Question

I'm quite new to cluster analysis and I was trying to perform a hierarchical clustering algorithm (in R) on my data to spot some groups in my dataset. Initially, I tried with the k-means, with the kmeans() functions, but the betweenss/totss that I found with k=4 was very low (around 28%), and also the trying with other little values of k the results were not satisfactory.Then I decided to try a hierarchical clustering algorithm.So I applied the function hclust() with three different methods and I found that the best linkage to use to split my data points into few groups is the average linkage (my aim was to have few groups to spot):

By observing the dendrogram, I considered the possibility to split my data into 4 clusters, and I plotted using the function fviz_dend():

I obtained this. How can I interpret this? How can I define what's the ideal number of clusters? How can explore the cluster to understand their specific characteristics? Are there specific R functions to use?Moreover, in the case of the k-means algorithm, I have the betweenss/totss that gives me the proportion of variance explained by the clusters rather than all my data points. Instead, what measures are useful in the case of hierarchical clustering?

Answer 1

The number of clusters problem is generally difficult and depends (as the problem of selecting a suitable clustering method) on the meaning of the data and the aim of clustering. Some methods produce better separated clusters that are less homogeneous, some others produce more homogeneous clusters that are less separated, and you may also be interested in other characteristics of clusters such as their distributional shape. Regarding the number of clusters, it is generally very hard to compare very different numbers of clusters (such as 4 vs. 20); most criteria are not well calibrated to do this,and anyway in some applications a large number of clusters (say 25 or 100 or even more) may be useless whereas in some other applications it may be required and splitting up the data into 2-4 clusters would be much to rough. What to do depends on what requirements a cluster needs to fulfill in your specific situation.

There are a number of criteria that can be optimised to find a good number of clusters such as the average silhouette width

https://www.sciencedirect.com/science/article/pii/0377042787901257

There are also a number of R-packages that have more such criteria, such as clValid, clusterCrit, and the cluster.stats function in fpc. Here several such criteria are listed:

https://cran.r-project.org/web/packages/clusterCrit/vignettes/clusterCrit.pdf

As I wrote, which one to use depends on your specific situation.

One possibility to explore and interpret the clusters once you have them is to look at the distribution of your variables within the clusters.

Answer 2

If you want to use your hierarchical chart to judge a good number of groups, then you can look at the height gap between splits, perhaps something like this.

Bigger gaps might be seen as better and narrow gaps as involving almost arbitrary choices.

So in this example, $5$ groups has a big gap, as does $15$ groups.

By contrast $7$, $9$, $12$ and $13$ groups have narrow gaps.

The other small numbers of groups are in a sense intermediate between these extremes.

This might guide rather than drive your decision on the number of groups. You might

• consider roughly how many groups you might use in terms of how much detail you want to work with,
• then avoid the nearby numbers of groups associated with very narrow gaps, and
• if there is wide gap nearby then use that,
• but if not then use an intermediate gap number close to what you originally considered