When I separate my data into groups and find the Pearson correlation coefficients between two features within each group, I get pretty high correlation values. However, the Pearson correlation value between the same two features for the whole dataset is so much smaller!
To visualize, I have a dataframe that looks something like this:
ID Val1 Val2
A .368 9026
A .393 12537
B .362 14511
B .366 21681
When I split into groups by the ID column and find the Pearson correlation between Val1 and Val2 within each group, I get
ID Correlation b/w Val1 and Val2
A 1.0
B 1.0
But when I calculate the Pearson correlation between the same features on the whole/original dataframe, I get .03.
Can someone help me understand why this happens? Does this mean one method is valid while the other isn't?