1
$\begingroup$

I have a set of scores on a specific dataset given by human evaluators, let's call it HJ. I also have the scores of two computer models on the same dataset, let's call these sets M1 and M2. I have calculated Spearman's coefficients for (HJ,M1) and (HJ,M2), which show that M2 is closer to human judgements than M1. Now I've been asked to find out if the difference between these two rho coefficients is statistically significant. My statistical knowledge is limited, so any help on how I could do that would be highly appreciated.

EDIT:

In the book "Applied Multiple Regression/Correlation Analysis for the Behavioural Sciences" (Cohen and Cohen, 1975), I found the following $t$-statistic formula for cases identical to the one I describe above, but for Pearson coefficients:

$$ t=\frac{(r_{XY}-r_{VY})\sqrt{(n-3)(1+r_{XV})}}{\sqrt{2(1-r^2_{XY}-r^2_{VY}-r^2_{XV}+2r_{XY} r_{XV} r_{VY})}} $$

where $r_{XY}$ is the correlation between $X$ and $Y$ and $n$ is the sample size. Is this also applicable to rank correlations? For a 2-tailed $t$-test it gives different results than the permutation test suggested by Greg below (in case I ask something trivial please excuse my ignorance).

$\endgroup$
3
$\begingroup$

One option is a permutation test. You would randomly swap within M1,M2 pairs then recompute the difference in the 2 correlations. Repeate that a bunch of times, then compare where your original difference lies compared to the permuted differences. Here is some R code to demonstrate:

> orig <- with(iris, cor(Sepal.Length, Petal.Length, method='s') - 
+ cor(Sepal.Length, Petal.Width, method='s') )
> 
> out <- replicate(999, {tmp <- rbinom(nrow(iris), 1, 0.5);
+ cor( iris$Sepal.Length, ifelse(tmp, iris$Petal.Length, iris$Petal.Width), method='s') -
+ cor( iris$Sepal.Length, ifelse(tmp, iris$Petal.Width, iris$Petal.Length), method='s')
+ })
> 
> out <- c(orig,out)
> hist(out)
> abline(v=orig, col='red')
> mean(  out >= orig )
[1] 0.239
> mean( abs(out) >= orig )
[1] 0.486
$\endgroup$
  • $\begingroup$ Hi and thank you for your answer. Let me see if I understood well. If M1=[m1,m2,m3,m4] and M2=[k1,k2,k3,k4], a permutation of this could be e.g. [k1,m2,m3,k4] and [m1,k2,k3,m4]? (I mean, the indices must always match, right?) Then we calculate rho for each one of those permuted sets with HJ set and we record the difference. Finally we calculate the percentage of those differences that are higher or equal to the original difference rho(HJ,M1)-rho(HJ,M2), which is our (one-sided) p-value. Is that correct? $\endgroup$ – dkar Apr 24 '12 at 20:39
  • $\begingroup$ Yes, everything is correct. Since the ultimate distribution is symetric (for every permutation you can switch everything and get the negative of the difference you computed) the 2-tailed test (compare absolute values) is probably more meaningful. $\endgroup$ – Greg Snow Apr 24 '12 at 21:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.