# What affects correlation in this situation?

Suppose a dataset I with 20 samples pairs of x and y collected from 20 different farms in State A.

Another dataset II with 20 samples pairs of x and y collected from 20 different farms in State A, B, C, and D.

In a hypothetical situation, x and y is significantly correlated in the dataset II, but not in x and y in dataset I.

My question: What factors might be causing the significant differnces in dataset I and dataset II?

I understand r = cov(x,y)/(sd_x *sd_y), but how do I go about thinking this result in terms of the formula for r?

• Don't think in terms of the formula: draw scatterplots instead. You will be able to see that there are no constraints on the relationships between the correlations in the two cases: any pair of values strictly between $-1$ and $1$ is possible.
– whuber
Sep 18 at 16:31
• Thank you for this note, @whuber. I am trying to understand how the std dev of x and y in the 2 datasets might affect correlation. If xy pairs come from more sites, logically it will have higher variability in dataset II compared to dataset I. So if the denominator of the r is higher due to higher variability (higher sd_x and sd_y), the value of r increases? How might cov(x,y) change in these two datasets? Sep 18 at 18:23
• See the question at stats.stackexchange.com/questions/567494 for an example of how to visualize things.
– whuber
Sep 18 at 19:29

I would have said a sample of $$20$$ observations rather than $$20$$ samples.

Suppose the four states are the four visible clusters in this scatterplot: In each state separately there is zero correlation, but in the data set as a whole there is significant correlation.

Also consider this one: In each of the four states there is a $$90\%$$ correlation with $$100$$ observations. In the four states combined there is not a high correlation.

• +1, but note that the case described by the op is the opposite case: each of the 4 point clouds are uncorrelated, but collectively they are correlated. Sep 18 at 22:44
• @knrumsey : I have now edited accordingly. Sep 19 at 22:38
• @MichaelHardy, this is exactly what I thought, and even presented to my examiner the exact same thing as to what might be happening. The lurking variable state is pretty evident, but they wanted a more 'mathematical' explanation for such behaviour--which I find none! Sep 19 at 23:09
• @RabinKC I think I would need to know a lot more about the context in order to say anything more about this. Sep 21 at 2:09
• @RabinKC : I'm guessing that what you mean is that the variance of the least-square estimator of the slope depends only on the variability of $x$ and not on that of $y$. The actual value of the estimator does depend on the $y$ values. Its variance does not. Sep 25 at 20:24

Don't think about "significant" vs. "not significant" (see Andrew Gelman's paper "The Difference between "Significant" and "not Significant" is not, itself, Statistically Significant". Instead, look at effect sizes.

If the effect sizes are very different, then that means that what is going on in states C and D is different from what is going on in A and B. Since you haven't told us what X and Y are, or what the four states are, we can guess. Since you mention farms, I'm guessing it's something to do with agriculture. So, perhaps states A and B are both small states with single climates, while C and D are large states with varied climates. Then restriction of range could be a reason for lower correlations in A and B.

Or, of course, it could be sampling error.