Significant p value but no correlation?

Question

I ran a correlation test on some data I have containing job satisfaction ratings (0-10) and annual salary numbers, and this was the result:

> cor.test(relevantSurveyResults$Salary, relevantSurveyResults$JobSatisfaction)

    Pearson's product-moment correlation

data:  relevantSurveyResults$Salary and relevantSurveyResults$JobSatisfaction
t = 4.8647, df = 5246, p-value = 1.18e-06
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.04002955 0.09389943
sample estimates:
       cor 
0.06701333 

All of my knowledge about this comes from googling, and I'm confused. On one hand, a small p value should indicate that there is a correlation, buton the other, the cor result is really close to zero, so there's no correlation?

What does this mean? Did I just use a wrong test?

Answer 1

There is a more complete discussion here.

The p-value and the r value measure very different things. Unfortunately, discussions on them for correlation can be confusing.

If p < alpha, you have good evidence to reject the null hypothesis that r = 0. But this says nothing about the size of r; only that it is not zero. The p-value also doesn’t tell us if the correlation is large enough to be of practical importance.

In your case, you have good evidence to reject the null hypothesis (p < 0.0001), and you have an estimate of the size of the correlation (r = 0.067). Cohen† would rate this r as less than “small”, but you shouldn’t consider this interpretation to be universal.

~ ~ ~

Practically speaking for your example there are a couple of things to think about.

First, as @Aksakal points out, because your data includes ordinal ratings (0 – 10 scale), you may want to consider rank-based tests like Spearman correlation or Kendall correlation.

Second, be sure to plot your data. Pearson correlation will work best for a linear relationship, and the rank-based correlations for monotonic relationships. If the data are curved in some other way, correlation may not capture the relationship well.

---

† Cohen, J. 1988. Statistical Power Analysis for the Behavioral Sciences, 2nd Edition. Routledge.

Answer 2

1. You can't trust these results too much because you have not accounted for confounding. Job satisfaction and salary co-vary as a function of several possible confounding variables such as the education of the employee, the gender or age of the employee, their household structure, the cost of living in the employee's city or town, tenure of employment, etc. Identifying such variables requires the analyst and investigator to sit down over many pots of coffee (or pints of beer) and hash out what variables are needed to make the right inference.

2. In a large sample, you have power to detect very small effects. A correlation of 0.06 may be true. Being close to 0 differs greatly from being 0, just as being very sick differs greatly from being dead.

3. We (statisticians) use "significance" with a great lack of precision. A correlation of 0.06 is statistically significant here, but it may not be practically significant or not noteworthy. You had too much precision, and you've detected something which honestly has no practical impact.