# Groupby Pearson's Correlation Coefficient vs Overall Pearson's Correlation Coefficient

When I separate my data into groups and find the Pearson correlation coefficients between two features within each group, I get pretty high correlation values. However, the Pearson correlation value between the same two features for the whole dataset is so much smaller!

To visualize, I have a dataframe that looks something like this:

ID  Val1    Val2
A   .368    9026
A   .393    12537
B   .362    14511
B   .366    21681


When I split into groups by the ID column and find the Pearson correlation between Val1 and Val2 within each group, I get

ID         Correlation b/w Val1 and Val2
A          1.0
B          1.0


But when I calculate the Pearson correlation between the same features on the whole/original dataframe, I get .03.

Can someone help me understand why this happens? Does this mean one method is valid while the other isn't?

Each pair of points happens to define a rising straight line uniquely, so the corresponding correlation is exactly 1. If each had defined a falling straight line uniquely, the correlation would have been exactly $$-$$1. (If there had been tied values on either variable for any data pair, the correlation would not be defined.)
For this example, I get a correlation for all data points of $$-$$0.278, but it seems clear that you have many more identifiers than 2. However, the explanation for correlations of magnitude 1 remains that they describe any configuration exactly matched by straight lines, as any introduction to correlation should explain. Otherwise what you mean by "pretty high" is not explained: $$\pm$$1 is as high as you can get.