Same Pearson correlation score

Question

Given the metrics A and B, applied on the same data, resulting in (normalized) scores, I use Pearson correlation between them and the gold standard. I get almost exactly the same Pearson correlation score for Pearson(A, gold-standard) and Pearson(B, gold-standard) which might at first glance mean that the two metrics perform the same (pearson: 0.5987 versus 0.5984).

I used paired t-test between the scores produced with A and the scores with B and it gives me "difference is considered to be extremely statistically significant". For more details of the result please see below.

Please help me understand why there is a statistical significant difference between the two metrics results, when their Pearson correlation to the gold standard is the same. Also please let me know how can I tell which metric outperforms the other one? Thank you!

Result of paired t-test: The mean of Group One minus Group Two equals 1.136 95% confidence interval of this difference: From 1.084 to 1.187

Edit after @tmrlvi commented:

Thank you very much for your reply. Actually I am not working with R. The pearson correlation script I use is available at the SemEval website (http://alt.qcri.org/semeval2017/task1/data/uploads/sts2017-trial-data.zip). As I figure out from my poor perl knowledge, they read the gold standard values in a vector and the system values in another vector.

Please consider these vectors, A is for scores derived using method A and B is for scores derived using method B. All values are between 0 and 5, with max 1 decimal.

A = (0.8, 1.3, 2.5, ...)

B = (1.2, 2.7, 3.1, ...)

gold-standard = (1, 1.4, 3, ...)

Using their script I got that Pearson(A, gs) = Pearson(B, gs) which made me think initially that the systems perform the same. But then using paired t-test(A,B) I got that there is a significant difference between them...I am trying to figure out which one outperforms and why the pearson score is the same when. I used a website for doing the t-test.

I think I am understanding a bit what you explained with your formula: i get the same Pearson score because given A, B is changing at every instance with almost the same step...like there is a relationship between (a_i, b_i) and (a_k, b_k).

My knowledge in statistics is quite poor, but I wish to understand what is happening here. Could you please emphasize on your explanation? Thank you.

Answer 1

Note that Pearson correlation measures how the centered variables correlate, that is how they change together at each observation. This is since, for all $a,b\in \mathbb{R}$,

$$ \frac{Cov\left(X,Y\right)}{\sqrt{Var\left(X\right)} \sqrt{Var\left(Y\right)}} = \frac{Cov\left(X + a,Y + b\right)}{\sqrt{Var\left(X + a\right)} \sqrt{Var\left(Y + b\right)}} $$ Therefore $ \rho\left(X,Y\right) = \rho\left(X + a,Y + b\right)$.

On the other hand, the paired t-test measures how significant is their difference. For any random variables $X$ and $X + d + \varepsilon $ (where $\varepsilon$ is also random) we have that if $\varepsilon$ is small and concentrated enough, then the two variable will have statistically significant difference in under paired t-test.

As an example, consider the vectors $\left(101, 100, 100, 100, \ldots \right)$ and $\left(1, 0, 0, 0, \ldots \right)$. They will have Pearson correlation of 1, but also unbounded $T$ statistic (as the standard deviation is zero), thus the difference will be statistically significant.

---

EDIT: Answering the follow-up questions

Pearson correlation is measures how much linear relationship there is between $A$ and $B$. That is, whether there is "relationship" between each $A_i$ and $B_i$.

Perhaps an illustration help drive the point home. I generated two vectors, $A$ and $B$, where $B$ is actually $A$ offsetted plus some noise. The golden standard is around the mean of $A$. All are plotted below.

Note how both $A$ and $B$ behave the same in each index - if $A$ rise, so does $B$. This behavior is reflected by their correlation - $\rho\left(A,B\right) \approx 1.0$. Note that this behavior doesn't hold for their relationship with $GS$, as is evident by the correlations $$ \rho\left(X, GS\right) \approx -0.107 \\ \rho\left(Y, GS\right) \approx -0.107$$

However, as you can also see from the graph, their differences are quite pronounced. Calculating the paired t-test gave me p-value of $3\times10^{-250}$ (for mean difference of $0.3$ and standard error of $9.4 \times 10^{-5}$). This is, without doubt, extremely unlikely. So, the difference is statistically significant.

Answer 2

EDIT: THE FOLLOWING ANSWERS THE CASE WHERE A AND B WERE NORMALIZED BY E.G. BY (A - min(A)) / (max(A) - min(A)) *5

The simple answer to your question is that Pearson correlation and paired t-test test very different things and report very different statistics.

The normalization process used --- e.g. (A - min(A)) / (max(A) - min(A)) *5 --- does not guarantee that the mean of each normalized group is the same. In the R code immediately below, the mean for A.std is 2.35, whereas the mean for B.std is 2.58. It is not surprising that the paired t-test finds a mean difference between groups statistically different from zero if the distributions of the data in the two groups are systematically different. In this example, (A - B) is negative or zero for every pair of data in A and B.

On the other hand, correlation assesses the degree to which A changes with a change in B, but isn't concerned with the absolute value of A vs. B. That is, (0, 1, 2, 3, 4) is perfectly correlated with (0, 100, 200, 300, 400), even though the second set of numbers is obviously larger. They correlate perfectly, but a t-test would find their means to be different. In the example immediately below, the correlations between A.std and Gold and B.std and Gold are nearly the same (r = 0.95).

Gold = c(1,   2,   2,   3, 3, 3, 4,   4,   5)
A =    c(3,   3,   3,   4, 4, 4, 5.0, 5.1, 5.2)
B =    c(0.0, 0.1, 0.2, 1, 1 ,1, 2,   2,   2)

A.std = (A - min(A))/(max(A) - min(A))*5
B.std = (B - min(B))/(max(B) - min(B))*5

round(A.std, 1)

  ### 0.0 0.0 0.0 2.3 2.3 2.3 4.5 4.8 5.0

round(B.std, 1)

  ### 0.0 0.2 0.5 2.5 2.5 2.5 5.0 5.0 5.0

mean(A.std)

  ### 2.35

mean(B.std)

  ### 2.58

cor.test(A.std, Gold)

  ### r = 0.952, p < 0.0001

cor.test(B.std, Gold)

  ### r = 0.953, p < 0.0001 

plot(A.std, Gold)
plot(B.std, Gold)

t.test(A.std, B.std, paired=TRUE)

  ### mean of differences = -0.235
  ### p = 0.003

hist(I(A.std-B.std)

EDIT: THE FOLLOWING ANSWERS THE CASE WHERE A AND B WERE NORMALIZED BY E.G. BY (A - mean(A) ) / sd(A).

This is a followup to the answer by @tmrlvi .

I thought it would be easy to come up with some reasonable toy data to satisfy the parameters of the question, especially following the example of @tmrlvi , but the paired t-test is very sensitive to the precise values used in A and B.

In the R example below, it seems that the values have to be just-so to yield a significant paired t-test. Specifically that there is essentially no variation in the paired differences, but that --- by rounding errors I suppose in my example --- R doesn't perceive this as precisely zero variation, otherwise the function would fail.

With a larger data set you might have a bit more in the values.

But I would recommend that the Original Poster consider their data carefully. Is there something non-random going on that makes the data meet the parameters of the question so precisely? Do the distributions of the data make Pearson correlation and t-test less desirable? Is there something undesirable going on with the normalization calculation?

Scatterplots and histograms of raw values, normalized values, and differences in paired values may be useful.

Gold = c(1,2,2,3,3,3,3,4,4,5)
A = c(103,102,101,100,100,100,100,100,100,100)
B = c(3,2,1,0,0,0,0,0,0,0)

A.std = (A - mean(A))/sd(A)
B.std = (B - mean(B))/sd(B)

cor.test(A.std, Gold)
cor.test(B.std, Gold)

plot(A.std, Gold)
plot(B.std, Gold)

t.test(A.std, B.std, paired=TRUE)

hist(I(A.std-B.std))