We want to be able to predict user ratings of the appeal of some objects. We have a hypothesized objective feature that may correlate with the user ratings. We want to test whether this objective feature correlates with the user ratings. Initially we imagined this would be done with a correlation or regression, but see the problem below.
Note: we are novices at statistics.
For discussion, suppose that there are 10 objects, each rated by 10 people, so there are 100 user ratings. Each of the objects is also rated by the objective features, so there are 10 objective scores. It is not possible to do a correlation between the 100 and 10 due to the differing numbers of numbers (100 vs 10).
One approach, call it "average ratings", would be to get an average rating for each object (averaged across multiple users), and then do a correlation with the objective measure. The correlation is then of 10 "points" each with 2 dimensions (averaged subjective, and objective).
The problem with this is that the user ratings themselves might (or might not) have a high variance, and after averaging, the variance in ratings is ignored by the correlation. If the ratings have very high variance compared to the difference in mean ratings between objects, then the correlation should be suspect.
What about another approach, call it "replicated objective measure". Instead of averaging the user ratings from 100 down to 10, replicate the objective measure from 10 up to 100, and then do the correlation. The correlation is then applied to 100 "points", each with two dimensions (the rating from a particular user, and a replicated objective score).
As a toy example, let's imagine there are just 3 objects, and 3 people doing ratings. Each of the 3 people rates each object. The ratings are
(1,2,3), (4,5,6), (2,3,4)
where the parentheses indicate the object. So the first object has ratings of (1,2,3). The first human rater has assigned the score 1 to the first object, 4 to the second object, and 2 to the third object. (Of course these numbers are chosen for convenience and are not realistic.)
The objective feature has assigned the scores 2.0, 6.0, 3.0 to the three objects. Now we want to see if these objective scores are linearly related to the human ratings.
The problem is that there are a different number of scores:
9 human scores: 1,2,3, 4,5,6, 2,3,4 3 machine scores: 2.0, 6.0, 3.0
so I do not think a correlation or regression measure can be directly applied (?).
In the average ratings approach we average the human scores for each object, (1,2,3)->2, etc., to obtain 3 average scores that can then be correlated against the 3 machine scores.
3 *average* human scores: 2, 5, 3 3 machine scores: 2., 6., 3.
The problem is that such a correlation using only the averages ignores the variances in the human ratings. For example, if the human ratings were instead
(4,0,2), (5,1,9), (6,3,0)
the means are the same as the example above, but the big spread of ratings for each object suggest that the larger mean of the second object might be an accident.
In the "replicated objective measure" approach, instead of averaging the human ratings from 9 numbers down to 3, we replicate the machine scores for each object thee times, obtaining 9 numbers:
9 human scores: 1,2,3, 4,5,6, 2,3,4 9 replicated machine scores: 2,2,2, 6,6,6, 3,3,3
These numbers can then be plugged into a regression or correlation.
Question: Is this "replicated objective measure" approach valid? Is there a better approach to this problem?