1
$\begingroup$

Consider I have a $n \times n$ distance matrix I want to use for hierarchical clustering, where the distance metric I use ranges from 0-1.

I also have a second $n \times n$ matrix that gives me the p-value corresponding to each pairwise distance.

The p-value is calculated based on permutation tests. Here, for each pairwise comparison, the two vectors are randomly filled with entries of the joint vectors, and the distance is calculated again. This is repeated 1000 times to get the null distribution of the pairwise distance.

The goal of hierarchical clustering is to obtain a dendrogram that can be compared to existing dendrograms (e.g. cophenetic dissimilarity matrices, etc..), independent of this data or approach, and to test if identified clusters are the same. For this, I want my dendrogram to be as robust as possible, so including p-value info would potentially increase robustness in my hierarchical clustering.

So when using the distance matrix for clustering, is there any (known) way to include the p-value information, e.g., as weights, or does it make sense to set all as non-significant entries (given a p-value threshold) to say a distance of 1, when constructing the dendrogram?

How else can the information of pairwise distance significance be incorporated to build a more robust dendrogram? E.g., For using average linkage, can p-values or 1/pvalue used as weights somehow?

P.S. (I only have the distance matrix and the p-value matrix, and I don't want to move to bootstrapped distance matrices based on the original data).

Improve this question
$\endgroup$
5
  • $\begingroup$ I think people will be able to answer your question easier if you provide more context in your question. For example, what is the goal of h-clustering? How were the p-values on distance calculated? $\endgroup$
    – Alex J
    Commented Mar 1 at 4:46
  • $\begingroup$ Thanks! I added a little more info, if you have a specific question about the context happy to include it! Although this question is on purpose a little broader and more general about can p-values of pairwise distances improve clustering approaches. $\endgroup$
    – Jmmer
    Commented Mar 1 at 5:27
  • $\begingroup$ Tbh I have nothing to contribute to this, but hopefully this will help someone more knowledgable answer the question $\endgroup$
    – Alex J
    Commented Mar 1 at 5:49
  • 1
    $\begingroup$ What does a p-value for a distance mean? A p-value corresponds to a hypothesis test, what is the hypothesis test here? $\endgroup$
    – George Savva
    Commented Mar 1 at 8:57
  • $\begingroup$ the p-value here quantifies how my pairwise distance based on the original data (i.e., my two vectors) compares to the null distribution assuming the vectors are drawn from the same sample (are just random draws from the joint vector). If p is small I assume the distance is larger (smaller) than I would expect if the two vectors were just drawn from a joint sample. $\endgroup$
    – Jmmer
    Commented Mar 1 at 17:25

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.