I'm quite new to cluster analysis and I was trying to perform a hierarchical clustering algorithm (in R) on my data to spot some groups in my dataset. Initially, I tried with the k-means, with the kmeans() functions, but the betweenss/totss that I found with k=4 was very low (around 28%), and also the trying with other little values of k the results were not satisfactory.Then I decided to try a hierarchical clustering algorithm.So I applied the function hclust() with three different methods and I found that the best linkage to use to split my data points into few groups is the average linkage (my aim was to have few groups to spot):
By observing the dendrogram, I considered the possibility to split my data into 4 clusters, and I plotted using the function fviz_dend():
I obtained this. How can I interpret this? How can I define what's the ideal number of clusters? How can explore the cluster to understand their specific characteristics? Are there specific R functions to use?Moreover, in the case of the k-means algorithm, I have the betweenss/totss that gives me the proportion of variance explained by the clusters rather than all my data points. Instead, what measures are useful in the case of hierarchical clustering?