I have a data set, with each variable taking multiple values on a nominal scale. Separate raters could rate a given unit using more than one value per variable. That is, there are multiple ratings per unit/rater.
How to proceed from here?
I am aware that there are extensions of kappas (Fleiss) to support multiple ratings per unit/subject. Basically, the extension suggests computing separate Kappas per scale value, and then forming an aggregate.
Are there alternative strategies? How is the combined kappa exactly computed, and the auxiliaries in this case: standard error, ...?
(Unfortunately, right now, I do not have access to a copy of Fleiss' "Statistical methods for rates and proportions" because our library moves to a new location :/ Ch. 18 contains the necessary details)
Besides: How would this apply to Krippendorff's alpha (which I am also considering)?