Inter-rater reliability or intraclass correlation for one subject

I'm looking for a way, in R, to get measures of agreement for each item. I have 32 ordinal items (we want separate statistics for each), and 66 raters.I have two subjects, but we want to look at the reliability within each subject, not across both of them.

I looked at kripp.alpha in the irr package, but it assumes multiple subjects and gives errors for a vector of item scores. The ICC function in the psych package also expects a matrix of subjects and judges.

How can we get a measure of agreement for the raters within each subject?

I just added a basic function called cat_per_object() to my agreement package on GitHub. This function will take in an object-by-rater matrix (in your case, object means items so it will be 32x66 for each subject separately), the possible categories or ordinal levels, and the type of weighting scheme to use to account for the ordered nature of the categories; it will then output the percent observed agreement across all raters for each object.

Here is an example applying this function to an example dataset in which 20 objects are assigned by 5 raters into categories {0, 1, 2, or 3} using linear weights.

# install.packages("devtools")
# devtools::install_github("jmgirard/agreement")
> library(agreement)
> data(ordered)
> cat_per_object(ordered, categories = c(0, 1, 2, 3), weighting = "linear")
#> # A tibble: 20 x 3
#>    Object Weighting Agreement
#>    <chr>  <chr>         <dbl>
#>  1 1      linear        0.778
#>  2 2      linear        0.833
#>  3 3      linear        0.833
#>  4 4      linear        1
#>  5 5      linear        1
#>  6 6      linear        1
#>  7 7      linear        0.667
#>  8 8      linear        0.556
#>  9 9      linear        1
#> 10 10     linear        0.833
#> 11 11     linear        0.778
#> 12 12     linear        1
#> 13 13     linear        0.833
#> 14 14     linear        0.8
#> 15 15     linear        1
#> 16 16     linear        1
#> 17 17     linear        0.833
#> 18 18     linear        0.611
#> 19 19     linear        0.833
#> 20 20     linear        1


You can see how this function works by viewing its source code.

• This is excellent! Thank you! It would be helpful to have some documentation of the fact that the function expects items as rows and raters as columns.
– hare
Nov 18 '19 at 18:00