Groupby Pearson's Correlation Coefficient vs Overall Pearson's Correlation Coefficient

Question

When I separate my data into groups and find the Pearson correlation coefficients between two features within each group, I get pretty high correlation values. However, the Pearson correlation value between the same two features for the whole dataset is so much smaller!

To visualize, I have a dataframe that looks something like this:

ID  Val1    Val2    
A   .368    9026       
A   .393    12537         
B   .362    14511       
B   .366    21681           

When I split into groups by the ID column and find the Pearson correlation between Val1 and Val2 within each group, I get

ID         Correlation b/w Val1 and Val2
A          1.0
B          1.0

But when I calculate the Pearson correlation between the same features on the whole/original dataframe, I get .03.

Can someone help me understand why this happens? Does this mean one method is valid while the other isn't?

Answer 1

The way to understand what is happening is to look at your data example and consider what correlation is measuring. Here is a plot:

Each pair of points happens to define a rising straight line uniquely, so the corresponding correlation is exactly 1. If each had defined a falling straight line uniquely, the correlation would have been exactly $-$1. (If there had been tied values on either variable for any data pair, the correlation would not be defined.)

The explanation doesn't depend on which variable goes on which axis.

For this example, I get a correlation for all data points of $-$0.278, but it seems clear that you have many more identifiers than 2. However, the explanation for correlations of magnitude 1 remains that they describe any configuration exactly matched by straight lines, as any introduction to correlation should explain. Otherwise what you mean by "pretty high" is not explained: $\pm$1 is as high as you can get.

More generally, the fact that groupwise results may contradict the total pattern of results is an amalgamation paradox often named for Edward H. Simpson, although it was known much earlier than when he wrote about it. Whether that is substantively or scientifically surprising or expected is something for the researcher to think about. Your problem description contains no detail that allows any suggestion there. The first step is almost always to look more carefully at the data: groupwise correlations being stronger or weaker than all-data correlations are both possible. Groups making no difference would imply that the grouping is irrelevant and uninformative.

Your correlations happen to be well-defined but they don't tell you very much.