Suppose I have a dataset with the following information:
- N objects, each of which can be rated
- The number of ratings per object is variable (raters are unidentified), e.g., there are 30 objects each with a unique id
- Each row has the predicted rating, the actual rating, and the difference between the two, which is an ordinal value, say between 0 and 10
How do I measure the consistency of ratings per object and for the whole dataset? Basically, there is a distribution of "differences" per object. What are the various ways to approach this problem? Thank you.
