Given the metrics A and B, applied on the same data, resulting in (normalized) scores, I use Pearson correlation between them and the gold standard. I get almost exactly the same Pearson correlation score for Pearson(A, gold-standard) and Pearson(B, gold-standard) which might at first glance mean that the two metrics perform the same (pearson: 0.5987 versus 0.5984).
I used paired t-test between the scores produced with A and the scores with B and it gives me "difference is considered to be extremely statistically significant". For more details of the result please see below.
Please help me understand why there is a statistical significant difference between the two metrics results, when their Pearson correlation to the gold standard is the same. Also please let me know how can I tell which metric outperforms the other one? Thank you!
Result of paired t-test: The mean of Group One minus Group Two equals 1.136 95% confidence interval of this difference: From 1.084 to 1.187
Edit after @tmrlvi commented:
Thank you very much for your reply. Actually I am not working with R. The pearson correlation script I use is available at the SemEval website (http://alt.qcri.org/semeval2017/task1/data/uploads/sts2017-trial-data.zip). As I figure out from my poor perl knowledge, they read the gold standard values in a vector and the system values in another vector.
Please consider these vectors, A is for scores derived using method A and B is for scores derived using method B. All values are between 0 and 5, with max 1 decimal.
A = (0.8, 1.3, 2.5, ...)
B = (1.2, 2.7, 3.1, ...)
gold-standard = (1, 1.4, 3, ...)
Using their script I got that Pearson(A, gs) = Pearson(B, gs) which made me think initially that the systems perform the same. But then using paired t-test(A,B) I got that there is a significant difference between them...I am trying to figure out which one outperforms and why the pearson score is the same when. I used a website for doing the t-test.
I think I am understanding a bit what you explained with your formula: i get the same Pearson score because given A, B is changing at every instance with almost the same step...like there is a relationship between (a_i, b_i) and (a_k, b_k).
My knowledge in statistics is quite poor, but I wish to understand what is happening here. Could you please emphasize on your explanation? Thank you.