14
$\begingroup$

(Thanks a lot for the quick responses! I did a poor job of asking the question, so let me retry.)

I do not know how to find out whether or not the difference between two Spearman's correlations is statistically significant. I would like to know how to find it out.

The reason I wanted to find out is that in the following paper: Wikipedia-based Semantic Interpretation for Natural Language Processing, by Gabrilovich and Markovitch (Journal of Artificial Intelligence Research 34 (2009) 443-498).

In Table 2 (p. 457), the authors are showing that their method (ESA-Wikipedia) achieves a higher and statistically significant Spearman's correlation than other methods, and I would like to do the same to show that my method is better than previous methods for some problem.

I do not know how they calculated statistical significance, and I would like to know. The author of the paper did state that Spearman's rank correlation was treated as Pearson's correlation. I am not sure if that is the right way to do it. I have two Spearman's correlations and I would like to know ifthe difference between them is statistically significant or not.

I am aware that web sites, such as http://faculty.vassar.edu/lowry/rdiff.html, provide online calculator for obtaining the difference between two Pearson's correlations. I am unable to find a similar online calculator for the difference between two Spearman's correlations.

A solution from the link provided by Peter Flom

NOTE: The procedures only support the Spearman's correlations that are under 0.6.

  1. Let $z_A$ = the Fisher transform of the observed correlation of set $A$, $z_B$ = the Fisher transform of the observed correlation of set $B$.

  2. For $i = 1,\dots,n$, let $y_{A_i} = nz_A- (n - 1)z_{A'i}$, where $z_{A'i}$ is the Fisher transform of set $A$ of the one-left-out correlation obtained by deleting $(x_i,y_i)$, re-ranking, and re-computing the correlation. (Each $z_{A'i}$ is based on $n-1$ pairs; each deletion is temporary, for that i only, not permanent.) Repeat for set $B$.

  3. $\bar y_A = \sum y_{A_i}/n$ is the jackknifed Fisher transform. Repeat for set $B$.

  4. $v_{\bar y_A} = \sum (y_{A_i}-\bar y_A)^2 /(n(n-1))$ is the variance of $\bar y_A$. Repeat for set $B$.

  5. Use a heteroscedastic (Welch-Satterthwaite) $t$-test to compare the two jackknifed estimates:

$$ t = \frac{\bar y_A - \bar y_B}{\sqrt{v_{\bar y_A} + v_{\bar y_B}}},\quad \text{df}=\frac{(v_{\bar y_A} + v_{\bar y_B})^2}{\frac{v_{\bar y_A}^2}{n_A-1}+\frac{v_{\bar y_B}^2}{n_B-1}}$$ where $n_A$ and $n_B$ are the number of samples of set $A$ and $B$ respectively.

Before first edit

I have got a human-rated set of ranking (HUMAN-RANKING), a set of ranking generated by the presently used, popular method (PRESENT-RANKING), and finally a set of ranking generated by my purposed method (MY-RANKING).

I calculated the Spearman's correlation between HUMAN-RANKING and PRESENT-RANKING. Let me call this: HUMAN-PRESENT-SPEARMAN.

I then found out the Spearman's correlation between HUMAN-RANKING and MY-RANKING. Let me call this: HUMAN-MY-SPEARMAN.

How can I find out if the difference between HUMAN-MY-SPEARMAN and HUMAN-PRESENT-SPEARMAN is statistically significant?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Welcome Patrick. I am struggling with the same issue but with Pearson r. If you check my entries, you will get a feel of what you can do. $\endgroup$
    – Adhesh Josh
    Commented Oct 29, 2011 at 7:48
  • $\begingroup$ Although you might have dificulty framing this question in statistical terms - it would be useful if we knew what exactly you were interested in. Are you interested in the closeness of the correlation (How closely the scores predict each other) or the existence of a relationship more than chance. Given that you appear to have ranked data, repeated in time it might be useful to do some reading on intra-class correlation coefficients. I hope I have that right, the question isn't completely clear. $\endgroup$
    – user6666
    Commented Oct 29, 2011 at 22:09
  • $\begingroup$ Thanks Adhesh and rosser. I am sorry for my poor description of my question. I have rewritten it. Hope it has become a understandable question. $\endgroup$
    – Patrick Chan
    Commented Oct 30, 2011 at 9:26
  • $\begingroup$ Hi! I am currently struggling with the same problem. Do you by any chance have a code ready that implements your suggestion? Also, why does it only work for correlation values below 0.6? $\endgroup$
    – fsociety
    Commented Oct 14, 2013 at 14:20

1 Answer 1

Reset to default
1
$\begingroup$

The paper you cite explains the method in the following terms:

[...] we show the statistical significance of the difference between the performance of ESA-Wikipedia (March 26, 2006) version) and that of other algorithms by using Fisher's z-transformation (Press, Teukolsky,Vetterling, & Flannery, Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press, 1997, Section 14.5).

I suggest you follow that reference, or have a look at the Wikipedia page on the Spearman coefficient for details.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Thanks Guillermo. I did suspect that they treated the Spearman's rank correlation as Pearson's correlation and calculated the difference of two Pearson's correlations. However, it seems to me it is not the correct way to do it, and so I am making a post here. $\endgroup$
    – Patrick Chan
    Commented Nov 21, 2011 at 4:08
  • $\begingroup$ Do you perhaps know of a working implementation (preferably on-line) because this is what the OP is after? $\endgroup$
    – chl
    Commented May 10, 2012 at 8:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.