I've gone ahead and clustered a dataset using a Euclidian Hierarchical Clustering algorithm:
from scipy import cluster distance_metric = 'euclidean' linkage_matrix = cluster.hierarchy.linkage(X_norm_not_missing, method='single', metric=distance_metric)
I'm then calculating the Cophenetic Coefficient in order to determine the goodness of fit of the clustering:
from scipy.spatial.distance import pdist cophenetic_corr_coef, _ = cluster.hierarchy.cophenet(linkage_matrix, pdist(X_norm_not_missing)) cophenetic_corr_coef
However, this calculates the values using the full hierarchical cluster, rather than a pruned one. When I go ahead and plot it, for example, I can specify a
p value to truncate the dendogram:
cluster.hierarchy.dendrogram(linkage_matrix, leaf_rotation=90, leaf_font_size=12, # no more than p levels of the dendogram tree are displayed truncate_mode='level', p=12, )
However, I'm not seeing a way to prune the actual model/linkage matrix, rather than simply the depiction of the dendogram of the linkage matrix. How can I go ahead and prune the hierarchical cluster that has been generated in order to calculate the Cophenetic Coefficient for a truncated Hierarchical Clustering algorithm?