I am interested in the comparison of Pearson correlation and Euclidean distance as measures of similarity between data points. Suppose I have 4 data points, w, x, y, z, in a multidimensional space, where w is a very extreme outlier and x, y, z are highly similar to each other. For the correlation measure, I assume that x, y, z have high correlation coefficients (positive) with each other, but not equal to 1. For the euclidean measure, I assume that x, y, z have small euclidean distances with each other, but not equal to 0. Now, if I take the arithmetic mean of w, x, y, z and call it m, how does the similarity between m and x, y, z change depending on the measure I use? Which measure is more robust to the presence of the outlier w and can still capture the similarity or closeness of x, y, z?
-
1$\begingroup$ The nature of "outlier" differs in the two circumstances, which makes it difficult to perform a direct comparison. If one is seeking a robust metric, neither of these choices would be near the top of the list. $\endgroup$– whuber ♦Commented Aug 3, 2023 at 15:34
2 Answers
Even without an outlier, correlation and closeness are fundamentally different things. Two things can correlate perfectly and be quite distant from each other. For a practical example, compare the strength in the dominant and non-dominant hand of a bunch of people. I bet there is very high correlation, but the dominant hand is a good bit stronger. Or look at the cost of living, over time, in New York City and Ottumwa, Iowa. Again, there will be high correlation but NYC is way more expensive.
Also, closeness is defined for pairs of points, but correlation is defined for pairs of variables. Unless I missed something, you don't have any variables, or, at least, you didn't name any.
However, if you do have some variables, then points that are outliers on a distance metric can be very different from those that are outliers on a correlation metric. An extreme point can be very close to the best fit line between the non-outliers.
-
$\begingroup$ Because the Pearson correlation coefficient is a measure of closeness on the spherical projection of the data, it's a stretch to characterize them as "fundamentally different." One might better characterize them as fundamentally the same with respect to all properties of the data modulo that projection. I don't think that would change your message, but it might help people understand it better. $\endgroup$– whuber ♦Commented Aug 3, 2023 at 16:50
$(1,2,3,4)$ and $(1000, 2000, 3000, 4000)$ have a perfect correlation of $1$, yet the Euclidean distance between them is rather large.
$(1,2,3,4)$ and $(1.1, 1.9, 2.7, 4.2)$ have am imperfect correlation but a much smaller Euclidean distance between them.
I would most certainly consider the second pair to be closer than the first.
In that sense, correlation is a rather poor measure of how close points are. Euclidean distance might be problematic when there is an outlier, so something like $\ell_1$ distance might be preferable, but I would not consider correlation for this.
Perhaps refer to my answer here. Other answers at that link are good, too. The images in my answer here might be helpful, too. I will include one here.
