How to interpret a hierarchical clustering dendrogram?

Question

I have a machine learning code that predicts whether a gene is likely to cause disease (the model gives the gene a score somewhere between 0 and 1, with 1 being causal for the disease).

The training data is 600 rows of genes with 8 features, I use the shap package to understand each feature's contribution to each genes output model prediction.

Each gene has a shap value per feature (measures degree of contribution each feature had on model prediction per gene) that I then use sqeuclidean hierarchical clustering on to view gene groupings. I have viewed this clustering with an interactive dendrogram, but I want to understand how to interpret this better.

A static view of my plot looks like this:

The x-axis is all the genes with their index numbers 0-600 (the graph is quite big so sorry for the image quality).

I have made an interactive plot in which you can hover over the final branches to see the specific gene name and the output machine learning model score like this:

What I want to understand is, when I hover my mouse over the top/upper branches I get one specific gene as text - I don't understand why this is. Is it because in this clustering method there are deciding genes that lead to the branch divisions further down?

The hover text over the higher branches that I don't understand how to interpret look like this:

Why is only 1 gene given in the hover text of the upper branches?

The code I use to get this hierarchical clustering is:

#1. Get shap values and run hierarchical clustering:

gb = GradientBoostingRegressor()
explainer = shap.Explainer(gb)
shap_values = explainer(X)
D = scipy.spatial.distance.pdist(shap_values.values[:,:-1], 'sqeuclidean')

ordered = scipy.cluster.hierarchy.complete(D)


#2. Get info for hover text in interactive plot:

gene_list = clusters["Gene"].tolist() #clusters is a pandas dataframe of Gene names and their scores
clusters = clusters.round(decimals=3)
clusters['title1'] = 'GB_Score:'
clusters['title2'] = 'Training_Score:'
clusters['title3'] = 'Gene:'

clusters2 = clusters[['title3', 'Gene', 'title1', 'GB_Score', 'title2', 'BPlabel_encoded']]

info = clusters2[clusters2.columns[0:]].apply(
    lambda x: ','.join(x.dropna().astype(str)),
    axis=1)

info = info.tolist()
info = [s.replace(':,', ': ') for s in info]
info = [s.replace(',', ', ') for s in info]

#3. Make interactive plot:

import plotly.figure_factory as FF
figure = FF.create_dendrogram(
    ordered, orientation='bottom', hovertext=info,
    linkagefun=lambda x: ordered
)

figure.update_layout({'width':1500, 'height':750},  font=dict(
        size=12
    )) 
figure.update_xaxes(visible=False, showticklabels=False)
   
figure.write_html("training_dendogram.html")

Edit:

Whilst I'm providing my specific example I'm stuck on, what I'm generally looking for is an understanding of how the upper branches make the decisions to be split further down - or does the dendrogram work from the bottom up and so potentially upper branches are like the final genes/rows to enter those groups? Any resources or any guidance to better understand this subject would be appreciated.

Answer 1

The higher branch is a cluster including all the genes that you can see below in that branch. The height of a node is the distance between the two subclusters/subbranches (how that distance is computed depends both on how you compute distance between single genes and how you aggregate distances in clusters, here I see you use complete linkage for that).

The reason for which the hover text only shows one gene is to be found in the code and in the code alone, because there's no theorical justification for that.