-1
$\begingroup$

I am checking correlations between arrays of values. These values are the weights of a network and some features of the nodes of the same network. I often get good correlations (sometimes > 0.6), but I always get an incredibly low p-value, usually between 1.0664e-48 and 9.6488e-181. So it's basically a 0, and this rings a few alarms in my mind, but I can't find an explanation.

How is that possible? How should I interpret these values?

EDIT: I'll try to explain myself better by simplifying the question. If I check the Pearson correlation (using a simple function in Python) between two vectors of length ~1300, and I get a good r value and an incredibly low p value, how should I interpret this?

I know what a p value is, but I never got such a low result. This is so low that it makes me wonder if there is something wrong in the analysis, so I was asking more experienced people what I could look for.

$\endgroup$
  • 1
    $\begingroup$ This doesn't make sense to me. Can you explain more about how you're computing the correlations and their p-values? $\endgroup$ – Glen_b Jul 9 '18 at 1:02
  • $\begingroup$ Question updated, I hope I clarified it. $\endgroup$ – sato Jul 9 '18 at 8:12
  • 3
    $\begingroup$ Ah. Thanks for clarifying. For some reason I had thought your samples were very small but they're very large. Nothing seems to be obviously astray here. What is the reason for your concern? $\endgroup$ – Glen_b Jul 9 '18 at 9:37
  • $\begingroup$ I thought there was something wrong, since I am still not used to analysis of very large data sample, so I needed opinions. Thanks a lot! $\endgroup$ – sato Jul 13 '18 at 8:29
2
$\begingroup$

First of all, a tiny p-value is not at all surprising for a moderate effect size with a large data set. Here's a quick brute-force example of the p-value of a random sample of 1300 values from a bivariate Normal with $\rho=0.6$:

> m <- MASS::mvrnorm(1300,mu=c(0,0),Sigma=matrix(c(1,0.6,0.6,1),2))
> cor.test(m[,1],m[,2])$p.value
[1] 1.562375e-128

For large data sets, it's quite common to get extremely low p-values. At these levels, you should remember that the interpretation of a p-value is conditional on all of the assumptions of the test (Normality, independence, etc.); the likelihood/degree of violation of some of those other assumptions is much higher than the p-value itself. (I sometimes tell students that the probability that some top-secret government agency broke in and secretly tampered with your data is much higher than the p-value.) Therefore, it's probably not worth taking the precise value of the p-value too seriously - this is one reason that the R language only reports exact p-values above a threshold of $2.2 \times 10^{-16}$.

One sometimes sees tiny p-values reported in bioinformatics where thousands or tens of thousands of replicate tests are being done (e.g. on a set of genetic markers); in this case, the p-value is really used as an index of the strength of evidence - still shouldn't be taken seriously as a precise quantitative value.

$\endgroup$
2
$\begingroup$

You want your p-value to be as low as possible for your value to be significant. If your p-value is above your significant level (usually 1% or 5%) then your correlation is not significant.

If you want to know more about the theory behind p-values, here is an easy explanation: https://www.medcalc.org/manual/correlation.php

$\endgroup$
0
$\begingroup$

Depending on what you are testing, you can think about p-values as a measure of how likely it was to get the result you obtained (or a more surprising result) from your test simply by chance. At 95% confidence, we decide to call any p values that are less than 0.05 'statistically significant'. If your p value is > 0.05, then your result is not statistically significant, ie., then your data was reasonably likely to occur even if the null hypothesis was true (i.e., there was an actual correlation of 0). Here is a nice summary that briefly mentions p-values for correlations. https://www.nature.com/news/statisticians-issue-warning-over-misuse-of-p-values-1.19503

The correlation (r-value) tells you how strong a relationship is between 2 variables. The p-value simply tells you how likely it was to get an r value as big (or bigger) as the one you actually got under the null hypothesis (i.e., assuming that there is a true population correlation of r = 0). Sometimes a large sample size will give you a significant (low) p-value with almost any slight difference between the "null" (little to no correlation between variables A & B) and the alternate (a significant correlation between variables A & B). In your case, it is reasonably unsurprising that a correlation > 0.6 would produce a significant p-value (ie, a p-value below 0.05, for a 95% confidence level) -- you would typically only see a strong correlation produce a non-significant result (ie, p > 0.05) if you have a very small sample size. In other words, the stronger your observed effect size (whether it's a correlation, Cohen's d, etc) the lower your p-value, all else being equal, while a smaller observed effect size will be accompanied by a larger p-value. If your sample size is large enough, even a "clinically non-significant" effect could still produce a p-value that indicates statistical significance.

https://www.ncbi.nlm.nih.gov/pubmed/26342862

$\endgroup$
  • $\begingroup$ The phrases "At 95% confidence" and "for a 95% confidence level" are at best redundant here. In fact they are best avoided: for example, there are significance tests not associated with confidence intervals and the idea of significance doesn't depend on a prior idea or choice of confidence level. Further, 0.05 is just a conventional level and hence is not the only way to decide on what should and should not be called significant. Also, your null, for your purposes, can't be "little or no correlation", just no correlation. $\endgroup$ – Nick Cox Jul 9 '18 at 7:55
  • $\begingroup$ Thanks a lot for the references. Actually the sample is quite large, even though "large" should be defined based on the context. Thanks for the hint anyway. $\endgroup$ – sato Jul 9 '18 at 8:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.