I have to compare two different lists of biomolecular annotations, and to find out their correlation degree. To do this, I have implemented a function computing the Spearman's rank correlation coefficient between two lists and a function computing the Kendall tau distance between two lists.

In my implementation, I have managed the case of having two lists of different dimensions and/or containing different elements, by this way: I take the elements of ListA that are missing in ListB and I insert them at the end of ListB; then I take the elements of ListB that are missing in ListA and I insert them at the end of ListA.

I thought this was a good way to proceed, but actually I discovered that it worked out only if the lists have similar dimensions (maximum difference: ~20%). Especially, if the two lists have very different dimensions, this approach give wrong results.

For example, I computed the two correlation metrics when ListA had dimension 4 and ListB had dimension 816. Obviously, these lists have little in common, but the Kendall and Spearman methods I implemented stated wrongly that they had a "high correlation". This happened because the 812 elements of ListB missing in ListA were inserted in ListA.

So the question is: how to manage correctly lists of different dimensions in computing Kendall and Spearman correlation coefficients?

How to manage the missing elements?

  • 1
    $\begingroup$ Surely only the observations present in both lists simultaneously tell you anything about the correlation, unless the missingness is somehow associated with size itself. Is this missingness at random? Completely at random? Something else? $\endgroup$ – Glen_b -Reinstate Monica May 28 '13 at 8:14
  • $\begingroup$ @Glen_b I cannot understand your comment. What do you mean for "missingness"? Do you mean "absence"? What do you intend for "at random"? The absence is related to the dataset we study. If a method predicts 4 annotations, and another predicts 816 annotations, we want to understand what is the similarity between these two predicted annotation lists. Hope this helps. $\endgroup$ – DavideChicco.it May 28 '13 at 12:41
  • $\begingroup$ No, it really doesn't clarify things. I apparently misunderstand your problem, but that doesn't clear it up at all. $\endgroup$ – Glen_b -Reinstate Monica May 28 '13 at 13:25
  • 1
    $\begingroup$ To have correlation you need to have several variables for each observation. That is, your samples need to be paired. Then, if one variable has less observations than others, it's because some data is missing. Anyway, if your samples are independent (not paired), then you can compute some kinds of distances, but that is not correlation. $\endgroup$ – Pere Oct 23 '16 at 20:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.