Evaluating correlation with multiple human annotators Assume we have 3 annotators, each one of which has assessed the quality of 3 products in a scale from 1 to 7. 
ANN  PRODUCT  SCORE
an1  pr1      5
an1  pr2      2
an1  pr3      3
an2  pr1      7
an2  pr2      1
an2  pr3      2
an3  pr1      3
an3  pr2      3
an3  pr3      4

We also have a computer model that makes predictions for the same products using a number of features. 
pr1  0.70
pr2  0.25
pr3  0.35

There are two ways to calculate the correlation of model's scores with human scores.


*

*First average the human scores, and then get the correlation with model's scores 
PRODUCT  ANN.SCORE  MODEL SCORE
pr1      (5+7+3)/3  0.70
pr2      (2+1+3)/3  0.25
pr3      (3+2+4)/3  0.35


*Repeat the model's score for every annotator and product, as follows:
ANN  PRODUCT  ANN.SCORE  MODEL SCORE
an1  pr1      5          0.70
an1  pr2      2          0.25
an1  pr3      3          0.35
an2  pr1      7          0.70
an2  pr2      1          0.25
an2  pr3      2          0.35
an3  pr1      3          0.70
an3  pr2      3          0.25
an3  pr3      4          0.35

and then get the correlation.
My question is, which method makes more sense from a statistical point of view? What are the actual differences between the two ways of measuring the correlation? 
 A: You are making very different hypothesis for the two cases. For the first case you get the correlation between the model score and the average human score, while for the second one you do not distinguish annotators, and compare the annotators value with the model score. In this case you are considering that the annotators are all the same, which does not make much sense from the practical point of view but, in my understanding, is still valid from a statistical point of view. What will probably happen is that since different annotators have different perceptions of quality, the points around linear relation will be much more scattered than for the first case, and the correlation coefficient will be smaller.
From a broader perspective, what you might consider is to try to understand how the different annotators value relates with the model score through regression analysis. You can do a multivariate linear regression (given the low number of annotators, you cannot go further than linear) on the model score using the annotators score (ann1, ann2, ann3). The score between the different annotators will be strongly correlated, but as seen in a different post, this does not constitute a problem; it is merely a multicollinearity issue, i.e., is equivalent to having a smaller number of measurements. 
A: First, maybe you should investigate the degree of agreement between the three human annotators. Search this site for agreement-statistics.  If the agreement is good, your two methods should give similar results. Otherwise, your method two should give some kind of average correlation, but you should investigate the reason for bad agreement. Also, plot your data, one plot for each human annotator.  
Maybe you could analyse with a linear mixed model, in R something like
lme4::lmer(ann_score ~ model_score + (model_score | ann), data=your_data_frame)

