Correlating perception with an objective measure - average ratings vs replicating the objective measure

Question

We want to be able to predict user ratings of the appeal of some objects. We have a hypothesized objective feature that may correlate with the user ratings. We want to test whether this objective feature correlates with the user ratings. Initially we imagined this would be done with a correlation or regression, but see the problem below.

Note: we are novices at statistics.

For discussion, suppose that there are 10 objects, each rated by 10 people, so there are 100 user ratings. Each of the objects is also rated by the objective features, so there are 10 objective scores. It is not possible to do a correlation between the 100 and 10 due to the differing numbers of numbers (100 vs 10).

One approach, call it "average ratings", would be to get an average rating for each object (averaged across multiple users), and then do a correlation with the objective measure. The correlation is then of 10 "points" each with 2 dimensions (averaged subjective, and objective).

The problem with this is that the user ratings themselves might (or might not) have a high variance, and after averaging, the variance in ratings is ignored by the correlation. If the ratings have very high variance compared to the difference in mean ratings between objects, then the correlation should be suspect.

What about another approach, call it "replicated objective measure". Instead of averaging the user ratings from 100 down to 10, replicate the objective measure from 10 up to 100, and then do the correlation. The correlation is then applied to 100 "points", each with two dimensions (the rating from a particular user, and a replicated objective score).

As a toy example, let's imagine there are just 3 objects, and 3 people doing ratings. Each of the 3 people rates each object. The ratings are

(1,2,3),   (4,5,6),    (2,3,4)

where the parentheses indicate the object. So the first object has ratings of (1,2,3). The first human rater has assigned the score 1 to the first object, 4 to the second object, and 2 to the third object. (Of course these numbers are chosen for convenience and are not realistic.)

The objective feature has assigned the scores 2.0, 6.0, 3.0 to the three objects. Now we want to see if these objective scores are linearly related to the human ratings.

The problem is that there are a different number of scores:

9 human scores: 1,2,3,   4,5,6,    2,3,4
3 machine scores:  2.0, 6.0, 3.0 

so I do not think a correlation or regression measure can be directly applied (?).

In the average ratings approach we average the human scores for each object, (1,2,3)->2, etc., to obtain 3 average scores that can then be correlated against the 3 machine scores.

3 *average* human scores: 2, 5, 3
3 machine scores:         2., 6., 3.

The problem is that such a correlation using only the averages ignores the variances in the human ratings. For example, if the human ratings were instead

(4,0,2),  (5,1,9),   (6,3,0)

the means are the same as the example above, but the big spread of ratings for each object suggest that the larger mean of the second object might be an accident.

In the "replicated objective measure" approach, instead of averaging the human ratings from 9 numbers down to 3, we replicate the machine scores for each object thee times, obtaining 9 numbers:

9 human scores:               1,2,3,  4,5,6, 2,3,4
9 replicated machine scores:  2,2,2,  6,6,6, 3,3,3

These numbers can then be plugged into a regression or correlation.

Question: Is this "replicated objective measure" approach valid? Is there a better approach to this problem?

Answer 1

I would suggest a different approach, a correlation with a constant or a factor does not really make sense. Instead you can use a regular linear model to test for differences in the user and objective scores.

So say you have 10 users each rating 10 different objects. For each object you also have an "objective" score, we assume this to be the true value. What you can do is subtract this true value from each user score. By doing that we "center" the user scores, and we will in effect test whether the "centered" scores are different from 0.

In running a linear model we will test if we can reject the null hypothesis that the "centered" scores are not different from 0 (for each object separately). If we can reject this hypothesis, that would mean that the users rated the objects differently to the true values, either higher or lower, depending on the sign of the coefficient.

If instead we cannot reject the null hypothesis that the "centered" scores are different from 0, that would mean that the users rated the objects similarly to true values.

To do that you simply take the difference of the user scores and the objective score. In my case I did that by "replicating" the true scores just for code simplicity, I could have done this just as well without "replicating" them, this was just for convenience.

Below is the code for the linear model using R.

> df=data.frame(user=c(1,2,3,4,5,6,2,3,4),
>               machine=c(2,2,2,6,6,6,3,3,3),
>               object=factor(c("A","A","A","B","B","B","C","C","C")))
> df$dif=df$user-df$machine
> mod=lm(dif~object,data=df)
> summary(mod)

Call:
lm(formula = dif ~ object, data = df)

Residuals:
   Min     1Q Median     3Q    Max 
    -1     -1      0      1      1 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept) -1.110e-16  5.774e-01   0.000    1.000
objectB     -1.000e+00  8.165e-01  -1.225    0.267
objectC      1.729e-16  8.165e-01   0.000    1.000

Residual standard error: 1 on 6 degrees of freedom
Multiple R-squared:   0.25, Adjusted R-squared:      0 
F-statistic:     1 on 2 and 6 DF,  p-value: 0.4219

The relevant part is in the coefficients table, where we get statistics for each object and p-values. Note: R uses reference contrasts, which means that the first term (Intercept) is the result for the first level of the categorical variable (A in this case, A is equal to 0), the other two terms are compared relative to the reference - A (B-A equals 0, and C-A equals 0).

From the results we see that we cannot reject any of the hypotheses (assume threshold at $\alpha=0.05$), that is A is not different from 0, while B and C are not different from A.

We are more interested in the differences for each object separately, that is A==0, B==0 and C==0. We can do that with the code below, the numbers in the K matrix define which values we sum, so c(1,0,0) means 1*A+0*(B-A)+0*(C-A)=A, c(1,1,0) means 1*A+1*(B-A)+0*(C-A)=B, ... This is just a specific of R.

> library(multcomp)
> K=rbind("A"=c(1,0,0),
>         "B"=c(1,1,0),
>         "C"=c(1,0,1))
> summary(glht(mod,linfct=K))

     Simultaneous Tests for General Linear Hypotheses

Fit: lm(formula = dif ~ object, data = df)

Linear Hypotheses:
           Estimate Std. Error t value Pr(>|t|)
A == 0 -1.110e-16  5.774e-01   0.000    1.000
B == 0 -1.000e+00  5.774e-01  -1.732    0.319
C == 0  6.189e-17  5.774e-01   0.000    1.000
(Adjusted p values reported -- single-step method)

The conclusions have not changed. We cannot reject any hypothesis for any object. Thus we cannot claim that the users rated the objects differently than the true scores.