I want to be able to calculate the agreement between two ratings measured at two separate times. The rating is on a scale of 1-5.
For example, the below data relates to 3 different "Rater"'s who measured different subjects "Subject" at two different times "Time.1" and "Time.2".
Ratings <- data.frame(Rater = c(1,1,1,2,2,2,3,3,3),Subject = 1:9, Time.1 = c(4,3,5,4,3,2,5,5,3), Time.2 = c(5,3,2,3,4,5,3,3,4))
I am assuming calculating a Weighted Kappa on just the Time.1 and Time.2 while ignoring the fact that the results come from different raters is taboo. So looking for alternative measures that can be implemented in R.