I am looking for a variant of Fleiss' Kappa to deal with interval data, rather than strictly nominal/ordinal data. The context that I intend to use it in is as follows:
- There are several (5-8) graders grading a total of 16 exams
- The exams are identical, and contain 7 questions.
- Each question is graded out of 3-8, depending on the question
- Every exam is graded by every grader (though there are spots of missing data)
Please help me find where to look! I have seen an occasional internet utterance of a weighted Fleiss' Kappa, but never a reference for it. I have hope that it exists because a weighted Cohen's Kappa is used frequently. References to relevant R packages would also be appreciated (I have used the irr package before).