1
$\begingroup$

I have a machine learning code that predicts whether a gene is likely to cause disease (the model gives the gene a score somewhere between 0 and 1, with 1 being causal for the disease).

The training data is 600 rows of genes with 8 features, I use the shap package to understand each feature's contribution to each genes output model prediction.

Each gene has a shap value per feature (measures degree of contribution each feature had on model prediction per gene) that I then use sqeuclidean hierarchical clustering on to view gene groupings. I have viewed this clustering with an interactive dendrogram, but I want to understand how to interpret this better.

A static view of my plot looks like this: enter image description here

The x-axis is all the genes with their index numbers 0-600 (the graph is quite big so sorry for the image quality).

I have made an interactive plot in which you can hover over the final branches to see the specific gene name and the output machine learning model score like this:

enter image description here

What I want to understand is, when I hover my mouse over the top/upper branches I get one specific gene as text - I don't understand why this is. Is it because in this clustering method there are deciding genes that lead to the branch divisions further down?

The hover text over the higher branches that I don't understand how to interpret look like this: enter image description here

Why is only 1 gene given in the hover text of the upper branches?

The code I use to get this hierarchical clustering is:

#1. Get shap values and run hierarchical clustering:

gb = GradientBoostingRegressor()
explainer = shap.Explainer(gb)
shap_values = explainer(X)
D = scipy.spatial.distance.pdist(shap_values.values[:,:-1], 'sqeuclidean')

ordered = scipy.cluster.hierarchy.complete(D)


#2. Get info for hover text in interactive plot:

gene_list = clusters["Gene"].tolist() #clusters is a pandas dataframe of Gene names and their scores
clusters = clusters.round(decimals=3)
clusters['title1'] = 'GB_Score:'
clusters['title2'] = 'Training_Score:'
clusters['title3'] = 'Gene:'

clusters2 = clusters[['title3', 'Gene', 'title1', 'GB_Score', 'title2', 'BPlabel_encoded']]

info = clusters2[clusters2.columns[0:]].apply(
    lambda x: ','.join(x.dropna().astype(str)),
    axis=1)

info = info.tolist()
info = [s.replace(':,', ': ') for s in info]
info = [s.replace(',', ', ') for s in info]

#3. Make interactive plot:

import plotly.figure_factory as FF
figure = FF.create_dendrogram(
    ordered, orientation='bottom', hovertext=info,
    linkagefun=lambda x: ordered
)

figure.update_layout({'width':1500, 'height':750},  font=dict(
        size=12
    )) 
figure.update_xaxes(visible=False, showticklabels=False)
   
figure.write_html("training_dendogram.html")

Edit:

Whilst I'm providing my specific example I'm stuck on, what I'm generally looking for is an understanding of how the upper branches make the decisions to be split further down - or does the dendrogram work from the bottom up and so potentially upper branches are like the final genes/rows to enter those groups? Any resources or any guidance to better understand this subject would be appreciated.

$\endgroup$
5
  • $\begingroup$ You’ll get better answers if (a) you tell us where FF came from, since the behavior seems to be inside that, and (b) if you provide a small sample of data enough to reproduce this behavior. $\endgroup$ Mar 27, 2021 at 16:52
  • $\begingroup$ Thanks for this, I didn't spot that I had left out my code for FF, will add that in. I can't give example data in this case, unfortunately - is there no reasoning from a theoretical perspective why hierarchical clustering would list specific samples in the upper branches? Ultimately I am looking for a general understanding as to how the upper branch splits are decided in this method. If it's more likely the problem is on the practical coding side of FF that would make things easier for me, but I haven't been able to find a good understanding of the theory of the clustering to know $\endgroup$
    – DN1
    Mar 27, 2021 at 18:31
  • 1
    $\begingroup$ One ought to read something about analysis methods they use. At minimum, start with wikipedia. Before asking questions. For example: ordered = scipy.cluster.hierarchy.complete(D) What does it do, basically? What is complete method? You must be able to answer this yourself. $\endgroup$
    – ttnphns
    Mar 27, 2021 at 19:43
  • $\begingroup$ Thank you for this advice. I have tried wikipedia and the scipy documentation for the functions, but I have a biology background so understanding particularly how the upper branches make decisions has eluded me - was hoping someone could give me a more general and accessible answer on the side of dendrogram interpretation here. $\endgroup$
    – DN1
    Mar 27, 2021 at 19:49
  • $\begingroup$ Complete method is among agglomerative clustering, not divisive (as you think). I goes bottom up. stats.stackexchange.com/q/195446/3277 $\endgroup$
    – ttnphns
    Mar 27, 2021 at 21:48

1 Answer 1

3
+50
$\begingroup$

The higher branch is a cluster including all the genes that you can see below in that branch. The height of a node is the distance between the two subclusters/subbranches (how that distance is computed depends both on how you compute distance between single genes and how you aggregate distances in clusters, here I see you use complete linkage for that).

The reason for which the hover text only shows one gene is to be found in the code and in the code alone, because there's no theorical justification for that.

$\endgroup$
1
  • $\begingroup$ Thank you so much, exactly what I was looking for but couldn't prove on my own understanding. $\endgroup$
    – DN1
    Mar 29, 2021 at 9:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.