# Hierarchical clustering and Dendrogram interpretation

I'm quite new to cluster analysis and I was trying to perform a hierarchical clustering algorithm (in R) on my data to spot some groups in my dataset. Initially, I tried with the k-means, with the kmeans() functions, but the betweenss/totss that I found with k=4 was very low (around 28%), and also the trying with other little values of k the results were not satisfactory.Then I decided to try a hierarchical clustering algorithm.So I applied the function hclust() with three different methods and I found that the best linkage to use to split my data points into few groups is the average linkage (my aim was to have few groups to spot):

By observing the dendrogram, I considered the possibility to split my data into 4 clusters, and I plotted using the function fviz_dend():

I obtained this. How can I interpret this? How can I define what's the ideal number of clusters? How can explore the cluster to understand their specific characteristics? Are there specific R functions to use?Moreover, in the case of the k-means algorithm, I have the betweenss/totss that gives me the proportion of variance explained by the clusters rather than all my data points. Instead, what measures are useful in the case of hierarchical clustering?

• I think you misinterpret the within-cluster sum of squares in k-means. This gives the within-cluster variance and should be small, because this means that the clusters are homogeneous. It is not a "percentage of variance explained" (actually it is not normally even a percentage, although one could standardise it to become one). – Lewian Nov 22 '20 at 20:47
• I had been vague. I was referring to the <<"Within cluster sum of squares by cluster" (between_SS / total_SS)>>. I read somewhere a similar interpretation of this index (variance explained), I could have gone wrong. – Cas Toyel Nov 22 '20 at 22:09
• I don't get that. Why is the between_SS in there? Do you mean you used the term within-cluster SS for the ratio "between_SS/total_SS"? Why would you do that? The "variance explained" interpretation is probably from Analysis of Variance and doesn't have implications for cluster analysis. – Lewian Nov 22 '20 at 22:18
• You must not select among hierarchical clustering methods by the looks of the dendrogram. If you are new to cluster analysis, please read the many questions here about "[clustering] number of clusters" (search). – ttnphns Nov 23 '20 at 5:52
• Most general overview of the guidelines to select a clustering method (as well as the number of clusters). Overview of the so-called internal clustering criteria aka internal validity indices. – ttnphns Nov 23 '20 at 5:57

The number of clusters problem is generally difficult and depends (as the problem of selecting a suitable clustering method) on the meaning of the data and the aim of clustering. Some methods produce better separated clusters that are less homogeneous, some others produce more homogeneous clusters that are less separated, and you may also be interested in other characteristics of clusters such as their distributional shape. Regarding the number of clusters, it is generally very hard to compare very different numbers of clusters (such as 4 vs. 20); most criteria are not well calibrated to do this,and anyway in some applications a large number of clusters (say 25 or 100 or even more) may be useless whereas in some other applications it may be required and splitting up the data into 2-4 clusters would be much to rough. What to do depends on what requirements a cluster needs to fulfill in your specific situation.

There are a number of criteria that can be optimised to find a good number of clusters such as the average silhouette width

https://www.sciencedirect.com/science/article/pii/0377042787901257

There are also a number of R-packages that have more such criteria, such as clValid, clusterCrit, and the cluster.stats function in fpc. Here several such criteria are listed:

https://cran.r-project.org/web/packages/clusterCrit/vignettes/clusterCrit.pdf

As I wrote, which one to use depends on your specific situation.

One possibility to explore and interpret the clusters once you have them is to look at the distribution of your variables within the clusters.

If you want to use your hierarchical chart to judge a good number of groups, then you can look at the height gap between splits, perhaps something like this.

Bigger gaps might be seen as better and narrow gaps as involving almost arbitrary choices.

So in this example, $$5$$ groups has a big gap, as does $$15$$ groups.

By contrast $$7$$, $$9$$, $$12$$ and $$13$$ groups have narrow gaps.

The other small numbers of groups are in a sense intermediate between these extremes.

This might guide rather than drive your decision on the number of groups. You might

• consider roughly how many groups you might use in terms of how much detail you want to work with,
• then avoid the nearby numbers of groups associated with very narrow gaps, and
• if there is wide gap nearby then use that,
• but if not then use an intermediate gap number close to what you originally considered