# Intuition behind pearson correlation, co-variance and cosine similarity

In this post, the best answer gives excellent mathmetical explanation among pearson correlation, co-variance and cosine similarity. Where I quote here ($\mathbf A$ is the data matrix).

• If you center columns (variables) of $\bf A$, then $\bf A'A$ is the scatter (or co-scatter, if to be rigorous) matrix and $\mathbf {A'A}/(n-1)$ is the covariance matrix.
• If you z-standardize columns of $\bf A$ (subtract the column mean and divide by the standard deviation), then $\mathbf {A'A}/(n-1)$ is the Pearson correlation matrix: correlation is covariance for standardized variables. The correlation is also called coefficient of linearity.
• If you unit-scale columns of $\bf A$ (bring their SS, sum-of-squares, to 1), then $\bf A'A$ is the cosine similarity matrix. Cosine is also called coefficient of proportionality.

In addition to math explanation, is there any intuitive plot such as pearson correlation in Wikipedia (shown below) to show the relationship between these three "similarity measures", i.e., what kind of shape each similarity metric is able to detect?

• The only way these differ substantially is that $A^\prime A$ includes information about magnitudes of the variables. If you don't include that as part of your notion of "similarity," then all three give identical information. It therefore sounds like your question has already been answered (unless you have some specific sense of "similarity" in mind). – whuber Jun 9 '16 at 16:42
• @whuber Thank you for reply. Are you saying they are identical and the only difference is they incorporate the magnitude differently? Is there any similar visualizations like the picture in Wikipedia for covariance and cosine similarity? – Haitao Du Jun 9 '16 at 16:55
• Such pictures would look identical to the one you re-posted, except possibly stretched or compressed vertically In fact, it is obvious that in the middle row the $x$ and $y$ variables have different spreads, indicating they are depicting the first case anyway. – whuber Jun 9 '16 at 17:43
• All three coefficients are about equal useless in measuring associations of the data shown in the last row. – ttnphns Jun 9 '16 at 18:27
• thanks @ttnphns. So for detecting associations in non-linear, which one is better, mutual information? – Haitao Du Jun 9 '16 at 18:34

We can ignore the matrix formulation, and just consider two vectors $x$ and $y$ (since the matrix formulation is just the vector operation repeated over different pairs of vectors). One intuitive/geometric distinction between covariance/correlation/cosine similarity is their invariance to different transformations of the input. That is, if we transform $x$ and $y$, under what types of transformations will the scores keep the same value?