0
$\begingroup$

Given the metrics A and B, applied on the same data, resulting in (normalized) scores, I use Pearson correlation between them and the gold standard. I get almost exactly the same Pearson correlation score for Pearson(A, gold-standard) and Pearson(B, gold-standard) which might at first glance mean that the two metrics perform the same (pearson: 0.5987 versus 0.5984).

I used paired t-test between the scores produced with A and the scores with B and it gives me "difference is considered to be extremely statistically significant". For more details of the result please see below.

Please help me understand why there is a statistical significant difference between the two metrics results, when their Pearson correlation to the gold standard is the same. Also please let me know how can I tell which metric outperforms the other one? Thank you!

Result of paired t-test: The mean of Group One minus Group Two equals 1.136 95% confidence interval of this difference: From 1.084 to 1.187

Edit after @tmrlvi commented:

Thank you very much for your reply. Actually I am not working with R. The pearson correlation script I use is available at the SemEval website (http://alt.qcri.org/semeval2017/task1/data/uploads/sts2017-trial-data.zip). As I figure out from my poor perl knowledge, they read the gold standard values in a vector and the system values in another vector.

Please consider these vectors, A is for scores derived using method A and B is for scores derived using method B. All values are between 0 and 5, with max 1 decimal.

A = (0.8, 1.3, 2.5, ...)

B = (1.2, 2.7, 3.1, ...)

gold-standard = (1, 1.4, 3, ...)

Using their script I got that Pearson(A, gs) = Pearson(B, gs) which made me think initially that the systems perform the same. But then using paired t-test(A,B) I got that there is a significant difference between them...I am trying to figure out which one outperforms and why the pearson score is the same when. I used a website for doing the t-test.

I think I am understanding a bit what you explained with your formula: i get the same Pearson score because given A, B is changing at every instance with almost the same step...like there is a relationship between (a_i, b_i) and (a_k, b_k).

My knowledge in statistics is quite poor, but I wish to understand what is happening here. Could you please emphasize on your explanation? Thank you.

$\endgroup$
  • $\begingroup$ Do I understand correctly? You performed the Pearson correlation with the normalized scores for A and B (range = 0 - 5), and the paired t-test used the normalized scores (range 0 - 5), and the mean difference from that test is 1.1 ? $\endgroup$ – Sal Mangiafico Nov 22 '17 at 16:59
  • $\begingroup$ @SalMangiafico: i computed the pearson between the gold standard and A, then the pearson between the gold standard and B and got almost the same number. The paired t-test was between A and B as I wanted to know if one of them is better. The mean difference is 1.136. And yes, everything was done with data normalized 0-5. $\endgroup$ – Tsuk Nov 22 '17 at 17:05
  • $\begingroup$ What formula did you use to normalize A and B? And what where approximate starting ranges of A and B? $\endgroup$ – Sal Mangiafico Nov 22 '17 at 17:10
  • $\begingroup$ @SalMangiafico: my scores were initially between 0 and 1, but I multiplied them by 5 in order to respect the SemEval request and to be able to use their script. I used scipy to get the initial scores for A and B and that had the normalization (0-1) by default. The initial A and B, before multiplying by 5, contained 0 values also. $\endgroup$ – Tsuk Nov 22 '17 at 17:15
  • $\begingroup$ I think the answer to your question rests on what you mean by "normalized". Usually, when we "normalize", we adjust the mean to a uniform value. For example, subtracting the mean and dividing the standard deviation yields a mean of zero and a standard deviation of 1 Wikipedia. In this case, it would be unusual for there to be a mean difference by t-test statistically diffrent thn 0, because the means of the two groups have been made to be the same. I believe it is under this assumption that the answers below were written. $\endgroup$ – Sal Mangiafico Nov 22 '17 at 17:39
0
$\begingroup$

Note that Pearson correlation measures how the centered variables correlate, that is how they change together at each observation. This is since, for all $a,b\in \mathbb{R}$,

$$ \frac{Cov\left(X,Y\right)}{\sqrt{Var\left(X\right)} \sqrt{Var\left(Y\right)}} = \frac{Cov\left(X + a,Y + b\right)}{\sqrt{Var\left(X + a\right)} \sqrt{Var\left(Y + b\right)}} $$ Therefore $ \rho\left(X,Y\right) = \rho\left(X + a,Y + b\right)$.

On the other hand, the paired t-test measures how significant is their difference. For any random variables $X$ and $X + d + \varepsilon $ (where $\varepsilon$ is also random) we have that if $\varepsilon$ is small and concentrated enough, then the two variable will have statistically significant difference in under paired t-test.

As an example, consider the vectors $\left(101, 100, 100, 100, \ldots \right)$ and $\left(1, 0, 0, 0, \ldots \right)$. They will have Pearson correlation of 1, but also unbounded $T$ statistic (as the standard deviation is zero), thus the difference will be statistically significant.


EDIT: Answering the follow-up questions

Pearson correlation is measures how much linear relationship there is between $A$ and $B$. That is, whether there is "relationship" between each $A_i$ and $B_i$.

Perhaps an illustration help drive the point home. I generated two vectors, $A$ and $B$, where $B$ is actually $A$ offsetted plus some noise. The golden standard is around the mean of $A$. All are plotted below.

Note how both $A$ and $B$ behave the same in each index - if $A$ rise, so does $B$. This behavior is reflected by their correlation - $\rho\left(A,B\right) \approx 1.0$. Note that this behavior doesn't hold for their relationship with $GS$, as is evident by the correlations $$ \rho\left(X, GS\right) \approx -0.107 \\ \rho\left(Y, GS\right) \approx -0.107$$

However, as you can also see from the graph, their differences are quite pronounced. Calculating the paired t-test gave me p-value of $3\times10^{-250}$ (for mean difference of $0.3$ and standard error of $9.4 \times 10^{-5}$). This is, without doubt, extremely unlikely. So, the difference is statistically significant.

enter image description here

$\endgroup$
  • $\begingroup$ I edited my post, could you please have a look? $\endgroup$ – Tsuk Nov 22 '17 at 16:40
  • $\begingroup$ Added an illustration as a follow-up of what could have happened. Hope it helps. $\endgroup$ – tmrlvi Nov 22 '17 at 18:10
  • $\begingroup$ thank you very much for explanations. Could you please provide your R code for the plot? Also, could you please let me know your opinion on how to evaluate better? So I have two vectors with values between (0,1) and a gold standard vector in the same range. How can I tell which method performs better than the other one and if their difference is statistical significant (other than t-test)? Thank you! $\endgroup$ – Tsuk Nov 23 '17 at 16:28
  • $\begingroup$ @Tsuk, this is a good question, but we don't have enough details to answer it. Instead of doing it back and forth, join us in the chat $\endgroup$ – tmrlvi Nov 23 '17 at 21:44
  • $\begingroup$ I am sorry but I can't write on chat because I do not have reputation min 20. My goal is to know which one of the two scores vectors fits better to the gold standard vector. Details: there is a set of n tuples of sentences that have a certain degree of semantic relatedness. There is a gold standard vector that gives for each tuple a score saying how similar the 2 sentences are. I implemented two methods to determine this similarity and want to figure out which one of the two is better. So given a tuple (s_1, s_2) that has its gold standard of 3, method A gave 2.5 and method B 4. Thank $\endgroup$ – Tsuk Nov 23 '17 at 22:03
0
$\begingroup$

EDIT: THE FOLLOWING ANSWERS THE CASE WHERE A AND B WERE NORMALIZED BY E.G. BY (A - min(A)) / (max(A) - min(A)) *5

The simple answer to your question is that Pearson correlation and paired t-test test very different things and report very different statistics.

The normalization process used --- e.g. (A - min(A)) / (max(A) - min(A)) *5 --- does not guarantee that the mean of each normalized group is the same. In the R code immediately below, the mean for A.std is 2.35, whereas the mean for B.std is 2.58. It is not surprising that the paired t-test finds a mean difference between groups statistically different from zero if the distributions of the data in the two groups are systematically different. In this example, (A - B) is negative or zero for every pair of data in A and B.

On the other hand, correlation assesses the degree to which A changes with a change in B, but isn't concerned with the absolute value of A vs. B. That is, (0, 1, 2, 3, 4) is perfectly correlated with (0, 100, 200, 300, 400), even though the second set of numbers is obviously larger. They correlate perfectly, but a t-test would find their means to be different. In the example immediately below, the correlations between A.std and Gold and B.std and Gold are nearly the same (r = 0.95).

Gold = c(1,   2,   2,   3, 3, 3, 4,   4,   5)
A =    c(3,   3,   3,   4, 4, 4, 5.0, 5.1, 5.2)
B =    c(0.0, 0.1, 0.2, 1, 1 ,1, 2,   2,   2)

A.std = (A - min(A))/(max(A) - min(A))*5
B.std = (B - min(B))/(max(B) - min(B))*5

round(A.std, 1)

  ### 0.0 0.0 0.0 2.3 2.3 2.3 4.5 4.8 5.0

round(B.std, 1)

  ### 0.0 0.2 0.5 2.5 2.5 2.5 5.0 5.0 5.0

mean(A.std)

  ### 2.35

mean(B.std)

  ### 2.58

cor.test(A.std, Gold)

  ### r = 0.952, p < 0.0001

cor.test(B.std, Gold)

  ### r = 0.953, p < 0.0001 

plot(A.std, Gold)
plot(B.std, Gold)

t.test(A.std, B.std, paired=TRUE)

  ### mean of differences = -0.235
  ### p = 0.003

hist(I(A.std-B.std)

EDIT: THE FOLLOWING ANSWERS THE CASE WHERE A AND B WERE NORMALIZED BY E.G. BY (A - mean(A) ) / sd(A).

This is a followup to the answer by @tmrlvi .

I thought it would be easy to come up with some reasonable toy data to satisfy the parameters of the question, especially following the example of @tmrlvi , but the paired t-test is very sensitive to the precise values used in A and B.

In the R example below, it seems that the values have to be just-so to yield a significant paired t-test. Specifically that there is essentially no variation in the paired differences, but that --- by rounding errors I suppose in my example --- R doesn't perceive this as precisely zero variation, otherwise the function would fail.

With a larger data set you might have a bit more in the values.

But I would recommend that the Original Poster consider their data carefully. Is there something non-random going on that makes the data meet the parameters of the question so precisely? Do the distributions of the data make Pearson correlation and t-test less desirable? Is there something undesirable going on with the normalization calculation?

Scatterplots and histograms of raw values, normalized values, and differences in paired values may be useful.

Gold = c(1,2,2,3,3,3,3,4,4,5)
A = c(103,102,101,100,100,100,100,100,100,100)
B = c(3,2,1,0,0,0,0,0,0,0)

A.std = (A - mean(A))/sd(A)
B.std = (B - mean(B))/sd(B)

cor.test(A.std, Gold)
cor.test(B.std, Gold)

plot(A.std, Gold)
plot(B.std, Gold)

t.test(A.std, B.std, paired=TRUE)

hist(I(A.std-B.std))
$\endgroup$
  • $\begingroup$ Thank you, I am not using R, but I will try to plot my data. I updated my post, please have a look. $\endgroup$ – Tsuk Nov 22 '17 at 16:41
  • $\begingroup$ To run the R code, you could just go to r-fiddle.org/# , paste the code in the lower window, and hit enter. It might be useful to see the plots from the toy data. $\endgroup$ – Sal Mangiafico Nov 22 '17 at 18:24
  • $\begingroup$ many thanks for your details. I think that in my setting using Pearson does not help much with ranking. Given two vectors with values between (0,1) and a gold standard vector in the same range, how to tell which method outperforms the other one and if their difference is statistical significant (other than t-test)? Thank you very much. $\endgroup$ – Tsuk Nov 23 '17 at 16:31
  • $\begingroup$ There are many different ways for things to be different or the same, and you really need to decide what it is you are trying to determine when you want to know if the two methods are different. 1) First question is, Why are "normalizing" the response in that way?; Does that your decision as to whether the methods are equally good or not? 2) You probably want to compare the "actual values" (Gold standard) to the "predicted values" (Method A or B). There are various measures of error and accuracy to do this ( rcompanion.org/handbook/G_14.html ). 3) The fact that the (cont...) $\endgroup$ – Sal Mangiafico Nov 24 '17 at 16:56
  • $\begingroup$ (...cont) paired t-test came out significant, with a difference > 1 for measurements on a 0 - 5 scale suggests there is a meaningful difference. But you want to explore the differences between A and B or between A or B and Gold. Plotting "actual" vs. "predicted" values is a start. You may also want to explore for which quantiles are different ( rcompanion.org/handbook/F_15.html , or by quantile regression). Does it matter to your purposes if the measurements vary only in their 90th percentiles? (...cont). $\endgroup$ – Sal Mangiafico Nov 24 '17 at 17:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.