Return to Answer

added 2004 characters in body

Source Link

edited Sep 15, 2019 at 19:39

So say you have 10 users each rating 10 different objects. For each object you also have an "objective" score, we assume this to be the true value. What you can do is subtract this true value from each user score. By doing that we "center" the user scores, and we will in effect test whether the "centered" scores are different from 0.

In running a linear model we will test if we can reject the null hypothesis that the "centered" scores are not different from 0 (for each object separately). If we can reject this hypothesis, that would mean that the users rated the objects differently to the true values, either higher or lower, depending on the sign of the coefficient.

If instead we cannot reject the null hypothesis that the "centered" scores are different from 0, that would mean that the users rated the objects similarly to true values.

To do that you simply take the difference of the user scores and the objective score. In my case I will performdid that by "replicating" the analysistrue scores just for code simplicity, I could have done this just as well without "replicating" them, this was just for convenience.

Below is the code for the linear model using R.

After thisThe relevant part is in the coefficients table, where we can get statistics for each object and p-values. Note: R uses reference contrasts, which means that the individual objects meanfirst term (Intercept) is the result for the first level of the categorical variable (A in this case, A is equal to 0), the other two terms are compared relative to the reference - A (B-A equals 0, and C-A equals 0).

From the results we see that we cannot reject any of the hypotheses (assume threshold at $\alpha=0.05$), that is A is not different from 0, while B and C are not different from A.

We are more interested in the differences for each object separately, that is A==0, B==0 and standard errorsC==0. We can do that with the code below, the numbers in the K matrix define which values we sum, so c(1,0,0) means 1A+0(B-A)+0*(C-A)=A, c(1,1,0) means 1A+1(B-A)+0*(C-A)=B, ... This is just a specific of R.

The conclusions have not changed. We cannot reject any hypothesis for any object. Thus we cannot claim that the users rated the objects differently than the true scores.

I would suggest a different approach, a correlation with a constant or a factor does not really make sense. Instead you can use a regular linear model to test for differences in the user and objective scores. To do that you simply take the difference of the user scores and the objective score. I will perform the analysis using R.

After this we can get the individual objects mean differences and standard errors

If instead we cannot reject the null hypothesis that the "centered" scores are different from 0, that would mean that the users rated the objects similarly to true values.

To do that you simply take the difference of the user scores and the objective score. In my case I did that by "replicating" the true scores just for code simplicity, I could have done this just as well without "replicating" them, this was just for convenience.

Below is the code for the linear model using R.

The relevant part is in the coefficients table, where we get statistics for each object and p-values. Note: R uses reference contrasts, which means that the first term (Intercept) is the result for the first level of the categorical variable (A in this case, A is equal to 0), the other two terms are compared relative to the reference - A (B-A equals 0, and C-A equals 0).

From the results we see that we cannot reject any of the hypotheses (assume threshold at $\alpha=0.05$), that is A is not different from 0, while B and C are not different from A.

We are more interested in the differences for each object separately, that is A==0, B==0 and C==0. We can do that with the code below, the numbers in the K matrix define which values we sum, so c(1,0,0) means 1A+0(B-A)+0*(C-A)=A, c(1,1,0) means 1A+1(B-A)+0*(C-A)=B, ... This is just a specific of R.

The conclusions have not changed. We cannot reject any hypothesis for any object. Thus we cannot claim that the users rated the objects differently than the true scores.

Source Link

created Feb 7, 2019 at 9:06

user2974951

> df=data.frame(user=c(1,2,3,4,5,6,2,3,4),
>               machine=c(2,2,2,6,6,6,3,3,3),
>               object=factor(c("A","A","A","B","B","B","C","C","C")))
> df$dif=df$user-df$machine
> mod=lm(dif~object,data=df)
> summary(mod)

Call:
lm(formula = dif ~ object, data = df)

Residuals:
   Min     1Q Median     3Q    Max 
    -1     -1      0      1      1 

Coefficients:
              Estimate Std. Error t value Pr(>|t|)
(Intercept) -1.110e-16  5.774e-01   0.000    1.000
objectB     -1.000e+00  8.165e-01  -1.225    0.267
objectC      1.729e-16  8.165e-01   0.000    1.000

Residual standard error: 1 on 6 degrees of freedom
Multiple R-squared:   0.25, Adjusted R-squared:      0 
F-statistic:     1 on 2 and 6 DF,  p-value: 0.4219

After this we can get the individual objects mean differences and standard errors

> library(multcomp)
> K=rbind("A"=c(1,0,0),
>         "B"=c(1,1,0),
>         "C"=c(1,0,1))
> summary(glht(mod,linfct=K))

     Simultaneous Tests for General Linear Hypotheses

Fit: lm(formula = dif ~ object, data = df)

Linear Hypotheses:
           Estimate Std. Error t value Pr(>|t|)
A == 0 -1.110e-16  5.774e-01   0.000    1.000
B == 0 -1.000e+00  5.774e-01  -1.732    0.319
C == 0  6.189e-17  5.774e-01   0.000    1.000
(Adjusted p values reported -- single-step method)