Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

(Thanks a lot for the quick responses! I did a poor job of asking the question, so let me retry.)

I do not know how to find out whether or not the difference between two Spearman's correlations is statistically significant. I would like to know how to find it out.

The reason I wanted to find out is that in the following paper: Wikipedia-based Semantic Interpretation for Natural Language Processing, by Gabrilovich and Markovitch (Journal of Artificial Intelligence Research 34 (2009) 443-498).

In Table 2 (p. 457), the authors are showing that their method (ESA-Wikipedia) achieves a higher and statistically significant Spearman's correlation than other methods, and I would like to do the same to show that my method is better than previous methods for some problem.

I do not know how they calculated statistical significance, and I would like to know. The author of the paper did state that Spearman's rank correlation was treated as Pearson's correlation. I am not sure if that is the right way to do it. I have two Spearman's correlations and I would like to know ifthe difference between them is statistically significant or not.

I am aware that web sites, such as http://faculty.vassar.edu/lowry/rdiff.html, provide online calculator for obtaining the difference between two Pearson's correlations. I am unable to find a similar online calculator for the difference between two Spearman's correlations.

A solution from the link provided by Peter Flom

NOTE: The procedures only support the Spearman's correlations that are under 0.6.

  1. Let $z_A$ = the Fisher transform of the observed correlation of set $A$, $z_B$ = the Fisher transform of the observed correlation of set $B$.

  2. For $i = 1,\dots,n$, let $y_{A_i} = nz_A- (n - 1)z_{A'i}$, where $z_{A'i}$ is the Fisher transform of set $A$ of the one-left-out correlation obtained by deleting $(x_i,y_i)$, re-ranking, and re-computing the correlation. (Each $z_{A'i}$ is based on $n-1$ pairs; each deletion is temporary, for that i only, not permanent.) Repeat for set $B$.

  3. $\bar y_A = \sum y_{A_i}/n$ is the jackknifed Fisher transform. Repeat for set $B$.

  4. $v_{\bar y_A} = \sum (y_{A_i}-\bar y_A)^2 /(n(n-1))$ is the variance of $\bar y_A$. Repeat for set $B$.

  5. Use a heteroscedastic (Welch-Satterthwaite) $t$-test to compare the two jackknifed estimates:

$$ t = \frac{\bar y_A - \bar y_B}{\sqrt{v_{\bar y_A} + v_{\bar y_B}}},\quad \text{df}=\frac{(v_{\bar y_A} + v_{\bar y_B})^2}{\frac{v_{\bar y_A}^2}{n_A-1}+\frac{v_{\bar y_B}^2}{n_B-1}}$$ where $n_A$ and $n_B$ are the number of samples of set $A$ and $B$ respectively.

Before first edit

I have got a human-rated set of ranking (HUMAN-RANKING), a set of ranking generated by the presently used, popular method (PRESENT-RANKING), and finally a set of ranking generated by my purposed method (MY-RANKING).

I calculated the Spearman's correlation between HUMAN-RANKING and PRESENT-RANKING. Let me call this: HUMAN-PRESENT-SPEARMAN.

I then found out the Spearman's correlation between HUMAN-RANKING and MY-RANKING. Let me call this: HUMAN-MY-SPEARMAN.

How can I find out if the difference between HUMAN-MY-SPEARMAN and HUMAN-PRESENT-SPEARMAN is statistically significant?

share|improve this question
2  
Welcome Patrick. I am struggling with the same issue but with Pearson r. If you check my entries, you will get a feel of what you can do. –  Adhesh Josh Oct 29 '11 at 7:48
    
Although you might have dificulty framing this question in statistical terms - it would be useful if we knew what exactly you were interested in. Are you interested in the closeness of the correlation (How closely the scores predict each other) or the existence of a relationship more than chance. Given that you appear to have ranked data, repeated in time it might be useful to do some reading on intra-class correlation coefficients. I hope I have that right, the question isn't completely clear. –  rosser Oct 29 '11 at 22:09
    
Thanks Adhesh and rosser. I am sorry for my poor description of my question. I have rewritten it. Hope it has become a understandable question. –  Patrick Chan Oct 30 '11 at 9:26
    
Hi! I am currently struggling with the same problem. Do you by any chance have a code ready that implements your suggestion? Also, why does it only work for correlation values below 0.6? –  ph_singer Oct 14 '13 at 14:20

2 Answers 2

The paper you cite explains the method in the following terms:

[...] we show the statistical significance of the difference between the performance of ESA-Wikipedia (March 26, 2006) version) and that of other algorithms by using Fisher's z-transformation (Press, Teukolsky,Vetterling, & Flannery, Numerical Recipes in C: The Art of Scientific Computing. Cambridge University Press, 1997, Section 14.5).

I suggest you follow that reference, or have a look at the Wikipedia page on the Spearman coefficient for details.

share|improve this answer
1  
Thanks Guillermo. I did suspect that they treated the Spearman's rank correlation as Pearson's correlation and calculated the difference of two Pearson's correlations. However, it seems to me it is not the correct way to do it, and so I am making a post here. –  Patrick Chan Nov 21 '11 at 4:08
    
Do you perhaps know of a working implementation (preferably on-line) because this is what the OP is after? –  chl May 10 '12 at 8:12

There is a long discussion of this question (with cautions and a solution) here

share|improve this answer

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

    
Peter, thanks for the link. I am not too familiar with Statistics and do not understand why a Fisher's transform helps. Is it okay for you to explain what the cautions are and what the solution is? –  Patrick Chan Oct 31 '11 at 15:04
1  
Fisher's transform helps because correlations are bounded by -1 and 1, and the transform transforms them into an unbounded variable that is close to normally distributed. –  Peter Flom Nov 1 '11 at 18:26
    
I think currently I have a lack of Statistics knowledge to understand the discussion of your link. I will beef up my knowledge and check out your link again. Thanks a lot, Peter! –  Patrick Chan Nov 2 '11 at 0:30
    
@Patrick Chan - If you do so and still run into stumbling points, I hope you'll post again here, describing the specific points that are causing you trouble. Somebody will be sure to try to help clarify. –  rolando2 Nov 19 '11 at 15:31
    
I am going to update the post for the solution from the Peter's link. rolando2, it will be helpful if you can check my post to see if I misunderstand something. –  Patrick Chan Nov 21 '11 at 4:19

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.