3
$\begingroup$

This is a follow up to a Stack Overflow question

I'm calculating the Spearman rank correlation coefficient between two vectors using corr in Matlab.

The two vectors represent the frequency of terms in a different types of document. For example, one type might be a webpage and the other might be a newspaper article. So each vector is 1 by n where n is the number of of terms in my vocabulary. In this case, n=1500. The reason that I am calculating rank correlation is I want to be able to say whether the vocabulary distribution differs between the different types of document. I normalize each vector, so that the rank corresponds to the percentage of documents each vocabulary term appears in.

From this calculation, I get rho = .8879 and p = 0

As I understand it, when p is small, the correlation is significant, but this is so extremely small that I am slightly concerned.

This is what my data distribution looks like on a loglog plot.

Data plot

From the Matlab documentation for corr, the p-value for Spearman is computed using permutation distributions.

Here is my understanding of how this calculation works, building on the Wikipedia article about permutation testing. Initially the correlation coefficient is calculated as the "observed value of the test statistic, T(obs)". Then both input sets are mixed together and all possible resampling of the mixed datapoints are tested for the correlation coefficient. The one-sided p-value of the test is calculated as the proportion of sampled permutations where correlation is greater than or equal to T(obs). The two-sided p-value of the test is the proportion where it was less than or equal to T(obs).

Therefore, to get a p-value of zero, I would need to get all of the correlation coefficients for the sampled permutations to either be greater than or all be less than T(obs). That seems extremely unlikely since my datapoints don't lie exactly on a line.

Am I correctly understanding the calculation? Is there an easier way to understand what the p-value is describing here?

Here is a link to the data on Dropbox, if you want to see if you get the same results.

Improve this question
$\endgroup$
1
  • $\begingroup$ There's some more-or-less relevant discussion (in particular regarding meaningfulness of extremely small p-values) here $\endgroup$
    – Glen_b
    Oct 2, 2015 at 0:32

2 Answers 2

Reset to default
1
$\begingroup$

The P value isn't zero. It is just very small and the program you use rounds it off to zero, rather than (correctly) saying it is less than some threshold.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Agreed. I'd add that I see no real reason not to use the Pearson correlation between your variables logged. I see a cut-off that's presumably related to the lowest possible frequency of occurrence. (I'd expect to see more evidence of some words occurring once, but perhaps there is multiple over-plotting.) But the cut-off doesn't make the Pearson correlation any more problematic. $\endgroup$
    – Nick Cox
    Apr 22, 2015 at 1:24
  • $\begingroup$ I'm glad to know that the value is not incorrect, but can you explain the meaning of having a very very low p-value other than the measure being significant? How does the calculation produce such a low number? $\endgroup$
    – 2cents
    Apr 22, 2015 at 18:24
  • $\begingroup$ @2cents You have lots of data that form a tight relationship. If the null hypothesis of no correlation were true, it would be very very very unlikely to randomly sample data points with such a strong correlation coefficient. So the P value is really really tiny. $\endgroup$
    – Harvey Motulsky
    Apr 22, 2015 at 22:55
1
$\begingroup$

The smallest normalized floating point number is this:

>> realmin
ans =
  2.2251e-308

For Pearson correlation on your data, the p-value is computed using a large-sample approximation based on a t-statistic as follows:

t =
  48.1326
>> p = 2*tcdf(-abs(t),n-2)
p =
  3.6052e-309

For Spearman's, the t-statistic and p-value are

t =
   75.6753
>> p = 2*tcdf(-abs(t),n-2)
p =
     0

So, unless there is a bug, the p-value is not zero but is smaller than double-precision numbers can represent. Even denormal values smaller than realmin don't go this small.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I'm glad to know that the value is not incorrect, but can you explain the meaning of having a very very low p-value other than the measure being significant? How does the calculation produce such a low number? $\endgroup$
    – 2cents
    Apr 22, 2015 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.