1
$\begingroup$

I want to be able to calculate the agreement between two ratings measured at two separate times. The rating is on a scale of 1-5.

For example, the below data relates to 3 different "Rater"'s who measured different subjects "Subject" at two different times "Time.1" and "Time.2".

Ratings <- data.frame(Rater = c(1,1,1,2,2,2,3,3,3),Subject = 1:9, Time.1 = c(4,3,5,4,3,2,5,5,3), Time.2 = c(5,3,2,3,4,5,3,3,4))

enter image description here

I am assuming calculating a Weighted Kappa on just the Time.1 and Time.2 while ignoring the fact that the results come from different raters is taboo. So looking for alternative measures that can be implemented in R.

Improve this question
$\endgroup$
4
  • $\begingroup$ What do you mean by "two different rating measurements?" Is the item measured at time 1 and time 2 the same thing, just a subject's measurement by the same rate and at two different points in time or are you measuring something else all together? $\endgroup$
    – StatsStudent
    Aug 19, 2020 at 0:22
  • 1
    $\begingroup$ I have updated the question for clarity. The item measured at time 1 and 2 is the same thing. $\endgroup$
    – JFG123
    Aug 19, 2020 at 0:29
  • $\begingroup$ Also, it appears that you don't have any rater measuring the same subject, so how would you have any inter-rater reliability? $\endgroup$
    – StatsStudent
    Aug 19, 2020 at 0:30
  • $\begingroup$ That is the point, I am not so much concerned with inter-rater reliability as in this study design there will never be the multiple raters rating the same subject. I am concerned with the agreement between Time.1 and Time.2 because the hypothesis is that they should be the same. If I ignore the fact that different raters have made the ratings I can easily calculate an agreement statistic e.g. weighted kappa/accuracy to give me a sense of how well they agree, but this ignores the different Rater component, which I feel is necessary to account for. $\endgroup$
    – JFG123
    Aug 19, 2020 at 0:35

1 Answer 1

Reset to default
0
$\begingroup$

I had a similar dataset and I found that usually test-retest reliability is reported per rater. So you would have to split your dataset in three parts and calculate the test-retest reliability for each rater separately.

If you found another way to do it please let me know.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.