0
$\begingroup$

I did Pearson correlation and I got this results:

    Pearson's product-moment correlation

data:  m$P.x and m$P.y
t = 2823.5, df = 852010, p-value < 2.2e-16
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.9502929 0.9507029
sample estimates:
  cor
0.9504983

The elements I got in he previous correlation test are the same as bellow plus additional ones (df is higher)

cor.test(t$P.x, t$P.y, method = "pearson", conf.level = 0.95)

    Pearson's product-moment correlation

data:  t$P.x and t$P.y
t = 2000, df = 2e+06, p-value <2e-16
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.801 0.801
sample estimates:
  cor 
0.801 

How is this difference in correlation explained by adding more "df"?

I am comparing here two set of p values. (m$P.x and m$P.y)

My data looks like this:

> dim(m)
[1] 852688      3
> head(m)
          SNP    P.x    P.y
1: rs10000012 0.7563 0.7563
2: rs10000013 0.7007 0.7324
3:  rs1000002 0.1870 0.2263
4: rs10000029 0.9078 0.9078
5:  rs1000003 0.6359 0.6359
6: rs10000033 0.7704 0.7162

...

$\endgroup$
1
$\begingroup$

The degrees of freedom for a correlation is just the number of complete data points (after excluding any missing data) minus two. Your first data set, m has $852,010 + 2 = 852,012$ points. The second, t, has around $2e^6 = 2,000,000$. You're doing the tests with different data (even if one is a subset of the other), so you're getting different results.

$\endgroup$
4
  • $\begingroup$ Thank you for getting back to me. Can you please tell me what does the smaller data set (with 852,012) increases correlation to 0.95? Is it because there is less noise in data or how that can be explained? $\endgroup$ – anamaria Sep 18 '20 at 14:58
  • $\begingroup$ if anything I was expecting that with less observations the correlation would be smaller but it is the opposite. How that can be explained? $\endgroup$ – anamaria Sep 18 '20 at 16:09
  • $\begingroup$ @anamaria The first set doesn't represent a random subset of the second set, so you may well expect the correlation to be different. .... but in any case, even if the subset were a random subset of the larger set, you wouldn't expect the correlation to be smaller; it could be either smaller or larger, by some small amount. $\endgroup$ – Glen_b Sep 20 '20 at 2:18
  • $\begingroup$ Hi Glen_b that you for getting back to me. Is there is any formula to show how that could be quantified? $\endgroup$ – anamaria Sep 20 '20 at 2:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.