1
$\begingroup$

I understand that correlation and distance are separate ideas, but I would really like to use something like Spearman's rank correlation in a calculation in calculating the distance for purposes of clustering.

Is this done regularly? can someone point me in the right direction?

For the record, I am currently using the built in dist function in R

The dist function mentions I can create a distance object out of a correlation method like this: 

dd <- as.dist((1 - cor(USJudgeRatings, method='spearman')/2))

However I'm hoping for some expert insight about this.

Improve this question
$\endgroup$
4
  • $\begingroup$ I had a co-worker apply the 1 - cor_matrix method (in Python) for a client we were consulting for who wanted to use k-means clustering on some click-bait feature for their social media data. It seemed to work fine for their purposes. I will ask though, what data are you trying to cluster and what are you trying to gain from the cluster analysis? $\endgroup$
    – Jon
    Commented Oct 20, 2016 at 0:30
  • $\begingroup$ Hey @Jon, I am comparing gene expression data collected using two different methods: The old style assay (microarray) reports its values in relative log2 fold change (where the denominator is either a control or the average across all samples). The new style assay (RNA-sesq) gives actual counts, which I have log transformed. these values are not directly comparable, even if I take some sort of fold change, but I was hoping that their rank ordering would be. $\endgroup$
    – kmace
    Commented Oct 20, 2016 at 5:20
  • $\begingroup$ That's interesting. I've always heard people using hierarchical clustering for microarray data. Is there a reason you're using k-means (instead of hierarchical) for your RNA-seq data? $\endgroup$
    – Jon
    Commented Oct 20, 2016 at 15:20
  • $\begingroup$ Also, what information would rank ordering give you for your clustering? $\endgroup$
    – Jon
    Commented Oct 20, 2016 at 16:20

1 Answer 1

Reset to default
1
$\begingroup$

As mentioned in my comment, I've only heard of agglomerative hierarchical clustering being applied to microarray data. With that said, here are some useful references.

Using correlation as distance metric (for hierarchical clustering)

where the distance matrix is defined to be $d = \sqrt{2(1 - r)}$ I would review Converting similarity matrix to (euclidean) distance matrix to understand why that definition makes sense.

Here is another resource that discusses the use of correlation as distances for hierarchical clustering purposes. http://research.stowers-institute.org/efg/R/Visualization/cor-cluster/

Here is a resource that discusses the use of correlation as distances http://ramet.elte.hu/~podani/3-Distance,%20similarity.pdf

Now, you want to use k-means with a Spearman Rank correlation matrix. I do not know how relevant that methodology is to your field, so I can't really comment much on that. However, I can ask, once you follow through with your methodology and come out with a clustering model, how will you assess how appropriate that model is for your data? And how will you compare it against other (similar k-means) models? These are appropriate questions you should address as you continue your analysis. Will you be able to use a goodness-of-fit statistic to evaluate and compare against other clustering models?

In the context of those questions, an approach I recommend is using a Gaussian mixed model hierarchical clustering model. In R, you can use the hc function for hierarchical clustering found in themclust package. The library uses Bayes factors estimated by BIC (they ignore the leading negative sign) to compare the various potential cluster models. This way you'll at minimum be able to say Model 1 is better than Model 2.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.