2
$\begingroup$

Assume we have 3 annotators, each one of which has assessed the quality of 3 products in a scale from 1 to 7. 

ANN  PRODUCT  SCORE
an1  pr1      5
an1  pr2      2
an1  pr3      3
an2  pr1      7
an2  pr2      1
an2  pr3      2
an3  pr1      3
an3  pr2      3
an3  pr3      4

We also have a computer model that makes predictions for the same products using a number of features. 

pr1  0.70
pr2  0.25
pr3  0.35

There are two ways to calculate the correlation of model's scores with human scores.

  1. First average the human scores, and then get the correlation with model's scores 

    PRODUCT  ANN.SCORE  MODEL SCORE
pr1      (5+7+3)/3  0.70
pr2      (2+1+3)/3  0.25
pr3      (3+2+4)/3  0.35

  2. Repeat the model's score for every annotator and product, as follows:

    ANN  PRODUCT  ANN.SCORE  MODEL SCORE
an1  pr1      5          0.70
an1  pr2      2          0.25
an1  pr3      3          0.35
an2  pr1      7          0.70
an2  pr2      1          0.25
an2  pr3      2          0.35
an3  pr1      3          0.70
an3  pr2      3          0.25
an3  pr3      4          0.35

    and then get the correlation.

My question is, which method makes more sense from a statistical point of view? What are the actual differences between the two ways of measuring the correlation?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ I find your reasoning hard to follow. You are searching the correlation between which parameters? Do you have information about how the model score is calculated (why do you call it regression)? $\endgroup$
    – pedrofigueira
    Commented May 21, 2014 at 15:50
  • $\begingroup$ Hi, I think the question is clear: I need the correlation between model's scores and human scores (the way I got the model scores is irrelevant). If we had only one human, the solution is trivial. Now we have more than one human, what is the appropriate way to get the correlation of all human scores with model scores? $\endgroup$
    – dkar
    Commented May 21, 2014 at 16:02
  • $\begingroup$ The question is anything but clear, because you describe two procedures that estimate two very different things. We need information that will help us figure out what you need to accomplish. $\endgroup$
    – whuber ♦
    Commented May 10, 2022 at 23:06

2 Answers 2

Reset to default
1
$\begingroup$

You are making very different hypothesis for the two cases. For the first case you get the correlation between the model score and the average human score, while for the second one you do not distinguish annotators, and compare the annotators value with the model score. In this case you are considering that the annotators are all the same, which does not make much sense from the practical point of view but, in my understanding, is still valid from a statistical point of view. What will probably happen is that since different annotators have different perceptions of quality, the points around linear relation will be much more scattered than for the first case, and the correlation coefficient will be smaller.

From a broader perspective, what you might consider is to try to understand how the different annotators value relates with the model score through regression analysis. You can do a multivariate linear regression (given the low number of annotators, you cannot go further than linear) on the model score using the annotators score (ann1, ann2, ann3). The score between the different annotators will be strongly correlated, but as seen in a different post, this does not constitute a problem; it is merely a multicollinearity issue, i.e., is equivalent to having a smaller number of measurements.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I don't follow why you think the second case does not distinguish annotators, can you explain a bit more? The annotators are enumerated sequentially instead of all together (as in the first case), but there are three distinct annotators nevertheless. I guess the right question to ask is in the case of multiple human annotators (so multiple "gold standards" for evaluation), what is the best way to aggregate them in order to evaluate the correlation of some model. I understand you are saying that first method (averaging the human judgements first) is more sensible, right? $\endgroup$
    – dkar
    Commented May 22, 2014 at 18:57
  • $\begingroup$ From what I understood, in the second case you correlate the annotator score with the model score. So you treat equally an annotation from ann1, ann2 and ann3; the info on the annotator number is lost. I think this is valid but not insightful, because different annotators can have different inclinations to vote one way or another. If that is the case, this be revealed by different slopes in a multivariate regression. $\endgroup$
    – pedrofigueira
    Commented May 22, 2014 at 22:19
0
$\begingroup$

First, maybe you should investigate the degree of agreement between the three human annotators. Search this site for . If the agreement is good, your two methods should give similar results. Otherwise, your method two should give some kind of average correlation, but you should investigate the reason for bad agreement. Also, plot your data, one plot for each human annotator.

Maybe you could analyse with a linear mixed model, in R something like

lme4::lmer(ann_score ~ model_score + (model_score | ann), data=your_data_frame)
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.