0
$\begingroup$

I'm using Pearson and Spearman correlation between predicted value and ground truth to evaluate the performance of my model (a deep neural network). On my first dataset the longer I train my model the better Pearson and Spearman correlation are, but on my second dataset meanwhile Pearson increase, Spearman decrease. How is that possible? If the linear correlation (Pearson) increase, the non-linear correlation (Spearman) should be increasing too? I don't know how to interpret those results.

EDIT:
I have a vector with real scores and a vector with predicted scores, I just a calculate the correlation between both of them at each step of the training. For the first test set both Pearson and Spearman increase during the training, while for the second test set Pearson increase and Spearman decrease over the training.

EDIT 2:
A lot of people don't understand what a PCA is or just don't even read what I wrote, so I deleted this graph to avoid confusion.

EDIT 3:
I'm calculating is the one between the orange (predicted semantic similarity score) and blue curves (semantic similarity based on human review) below :
First testSet
corr1
Second testSet
corr2
mains differences between both test sets are the number of samples (4 times higher in second) and the score distribution.
For both testSets Pearson correlation increase during training:
Pearson
But for the second test set Spearman correlation decrease:
Spearman
I should have posted those graphs earlier. By the way, the difference in performance between both test set isn't the problem, it's really the fact that Spearman correlation decrease while Pearson correlation increase for the second test set that bothers me.

PS:
I asked this question here in the first place, but I didn't get any answer

|cite|improve this question
$\endgroup$
  • 1
    $\begingroup$ It's difficult to answer without more details about the data... My best guess is that you have outliers. $\endgroup$ – Vincent Guillemot May 18 '18 at 7:35
  • $\begingroup$ Looking at your plot provided as I was writing my original answer, it looks like the spearman correlation would be close to 0. Is it significantly different to 0 and is it significantly different between Pearson and Spearman? Could you also state the correlation values? Also, is the value stable between training and test sets? $\endgroup$ – ReneBt May 18 '18 at 9:12
  • $\begingroup$ could you provide a lot more explantion of your figures? What is 'performance' in the Y axis of the second two plots of edit3? What are the Y axes of the first 2? Neither seems to correspond to the correlation that would be extracted from the scatterplot provided in edit 2, which I would expect to be be close to 0 for Spearman's and only weakly positive for Pearson's. If you want a coherent explanation of your data you will need a coherent presentation. $\endgroup$ – ReneBt May 18 '18 at 10:03
  • $\begingroup$ Performance is the Pearson/Spearman correlation, I NEVER COMPUTE THE CORRELATION OF THE SCATTERPLOT, the correlation is from the 2nd and 3rd graph, where y is a score given by a human (0, 1, 2 or 3) for the blue curve and for the orange it's the cosinus similarity between embeddings computed by my model. It's the correlation between those 2 curves that I calculate. You can see the result at each step of the training in graph 4 and 5. Graph 2 and 3 show results at the end of the training. $\endgroup$ – François MENTEC May 18 '18 at 12:11
  • 1
    $\begingroup$ Your question is becoming murkier. I am still guessing that your first scatter plot shows predicted value and ground truth, but (1) you don't label your axes, which every school child should know! (2) you now seem to be saying that the correlations don't relate to the scatter plot at all. $\endgroup$ – Nick Cox May 18 '18 at 14:52
1
$\begingroup$

Pearson and Spearman correlations use the same formula, just with different inputs.

$$\text{corr} = \text{cov}(x,y)\ /\ \sigma(x)\sigma(y)$$

Here $x$ and $y$ are values for Pearson and ranks for Spearman.

If you have an extreme value (in either variable) then in

  1. Pearson correlation: You will see these values dominate the calculation. Values an order of magnitude higher will make a numerically larger contribution and if it happens that both variables are on the same side of the average for their axes it will make the correlation appear positive (falsely if the extreme values are a spurious artefact). But if one is high and the other below average or vice versa it will become negative.

  2. Spearman correlation: You will not see these values dominate the calculation. Values an order of magnitude higher cannot make a numerically outsize contribution because ranks are constrained in their values. If it happens that both variables are higher than average for their axes it will only perturb the correlation moderately, so this correlation is less sensitive to outliers.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I understand the difference between both measure, but I don't know how to explain/interpret that Pearson increase while Spearman decrease. $\endgroup$ – François MENTEC May 18 '18 at 9:42
  • $\begingroup$ your scatterplot in edit 2 visually looks like a weak pearson correlation leveraged by two outliers in the upper right. This would mean subtle effects could easily tip from negative to positive. To help you understand plot the ranks in the same way, these two outliers will not be so evident.Then t complete your understanding fit best fit linear lines to each. You should see the Pearson with a slight positive slope and the Spearman with a slight negative one. Either that or a calculation is wrong somewhere. $\endgroup$ – ReneBt May 18 '18 at 9:58
  • 2
    $\begingroup$ Good explanation (+1), but the word falsely is unjustified. You have explained how the measures work. The results are only false if the data are and we have no information on that. $\endgroup$ – Nick Cox May 18 '18 at 11:20
  • $\begingroup$ @NickCox a valid point, I've updated to qualify the use of falsely. I felt it worth retaining in some form as the point of outlier analysis is assessing for the risk of over leveraged models that may be based on spurious artefacts or random noise . $\endgroup$ – ReneBt May 18 '18 at 14:33
  • 1
    $\begingroup$ Thanks. I am going to stop here, but a word like spurious raises the same issue. That is, Pearson is sensitive to data points on the edges of the distribution. That's a statistical fact; whether a researcher wants to judge them false or spurious or otherwise hard to credit is another matter. In my field outliers are usually entirely genuine. $\endgroup$ – Nick Cox May 18 '18 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.