1
$\begingroup$

I'm looking for a way, in R, to get measures of agreement for each item. I have 32 ordinal items (we want separate statistics for each), and 66 raters.I have two subjects, but we want to look at the reliability within each subject, not across both of them.

I looked at kripp.alpha in the irr package, but it assumes multiple subjects and gives errors for a vector of item scores. The ICC function in the psych package also expects a matrix of subjects and judges.

How can we get a measure of agreement for the raters within each subject?

$\endgroup$

1 Answer 1

2
$\begingroup$

I just added a basic function called cat_per_object() to my agreement package on GitHub. This function will take in an object-by-rater matrix (in your case, object means items so it will be 32x66 for each subject separately), the possible categories or ordinal levels, and the type of weighting scheme to use to account for the ordered nature of the categories; it will then output the percent observed agreement across all raters for each object.

Here is an example applying this function to an example dataset in which 20 objects are assigned by 5 raters into categories {0, 1, 2, or 3} using linear weights.

# install.packages("devtools")
# devtools::install_github("jmgirard/agreement")
> library(agreement)
> data(ordered)
> cat_per_object(ordered, categories = c(0, 1, 2, 3), weighting = "linear")
#> # A tibble: 20 x 3
#>    Object Weighting Agreement
#>    <chr>  <chr>         <dbl>
#>  1 1      linear        0.778
#>  2 2      linear        0.833
#>  3 3      linear        0.833
#>  4 4      linear        1    
#>  5 5      linear        1    
#>  6 6      linear        1    
#>  7 7      linear        0.667
#>  8 8      linear        0.556
#>  9 9      linear        1    
#> 10 10     linear        0.833
#> 11 11     linear        0.778
#> 12 12     linear        1    
#> 13 13     linear        0.833
#> 14 14     linear        0.8  
#> 15 15     linear        1    
#> 16 16     linear        1    
#> 17 17     linear        0.833
#> 18 18     linear        0.611
#> 19 19     linear        0.833
#> 20 20     linear        1  

You can see how this function works by viewing its source code.

$\endgroup$
4
  • 1
    $\begingroup$ This is excellent! Thank you! It would be helpful to have some documentation of the fact that the function expects items as rows and raters as columns. $\endgroup$
    – hare
    Commented Nov 18, 2019 at 18:00
  • $\begingroup$ Thank you for creating this package and the cat_per_object function. How does one cite this particular agreement metric? Is it a kappa or alpha or Scott's pi? Please let us know how it should be called so we can properly attribute the reference. $\endgroup$ Commented Dec 26, 2022 at 18:45
  • $\begingroup$ @the_darkside Most chance-adjusted indexes (e.g., kappa, pi, gamma) use the same approach to estimating observed agreement (though Krippendorff's alpha uses a slightly different, and in my opinion, slightly worse approach). I used that standard approach here. The main difference between the various indexes is how they estimate chance agreement. $\endgroup$ Commented Jan 3, 2023 at 21:39
  • $\begingroup$ @the_darkside I would just call it "linearly weighted observed agreement index" or "quadratically weighted observed agreement index." $\endgroup$ Commented Jan 12, 2023 at 22:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.