I have a data set, with each variable taking multiple values on a nominal scale. Separate raters could rate a given unit using more than one value per variable. That is, there are multiple ratings per unit/rater.

How to proceed from here?

I am aware that there are extensions of kappas (Fleiss) to support multiple ratings per unit/subject. Basically, the extension suggests computing separate Kappas per scale value, and then forming an aggregate.

Are there alternative strategies? How is the combined kappa exactly computed, and the auxiliaries in this case: standard error, ...?

(Unfortunately, right now, I do not have access to a copy of Fleiss' "Statistical methods for rates and proportions" because our library moves to a new location :/ Ch. 18 contains the necessary details)

Besides: How would this apply to Krippendorff's alpha (which I am also considering)?

  • $\begingroup$ I can likely provide an answer if you can clarify a few points. Using the following language, please describe your rating task. For example, let's say raters A, B, and C taste ten brands of salsa and rate each brand on spiciness (mild, medium, or hot) and flavor (tangy, sweet, or smokey). You could describe this as three raters assign ten objects to one of three categories in two category sets: one ordinal (spiciness) and one nominal (flavor). $\endgroup$ – Jeffrey Girard Oct 17 '16 at 19:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.