1
$\begingroup$

I have a data set composed by 2.000.000 projects, each of them is defined by the number of active users and the size (KBs) of the project.

I would like to check whether a correlation exists between these two features. Does it make sense to use the Spearman's rank correlation coefficent over the entire population?

thanks in advance

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ You can measure something over a population. However, if that's really the population to which you want your inference to apply then there's no need to test it. It is what it is. $\endgroup$
    – Glen_b
    Commented Apr 20, 2014 at 0:12
  • $\begingroup$ thanks for answering, in my case I'm not interested in knowing the p-value coming from the Spearman's rank correlation, but I need just a number to measure/quantify the correlation. Basically, i'm doing this in R, I'm using the function 'cor' that returns just the correlation value, instead of using 'cor.test' that returns the correlation value plus the corresponding p-value. My only concern is whether this is correct or I should just plot the two variables $\endgroup$
    – user2314405
    Commented Apr 20, 2014 at 7:45
  • 3
    $\begingroup$ If the value of the Spearman correlation is meaningful for you, then certainly calculate it. (If you're just after some general measure of monotonic association, you might like to consider the Kendall correlation which might be more intuitive in interpretation). $\endgroup$
    – Glen_b
    Commented Apr 20, 2014 at 8:25

1 Answer 1

Reset to default
1
$\begingroup$

Yes. Spearman rank correlation relieves you from the burden of knowing the marginal distribution of the number of users and sizes. It requires you to convert the number of users and project sizes into ranks first, so you will be get 2 million pairs of ranking orders as the input. Then the formula basically computes a "distance" between the two permutations determined by these ranking orders. You can then test the hypothesis that these two quantities are correlated through p-value calculation based on the null distribution, that is, the distribution of Spearman's rank correlation when one of the ranking is just 1,2,3,...,2million, and the other one is uniformly random. It converges to some Gaussian distribution as the size goes to infinity, which makes the calculation easy. These calculations and other related ranking distance functions can be found in this paper by Diaconis and Graham.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.