Suppose I have a dataset with the following information:

  • N objects, each of which can be rated
  • The number of ratings per object is variable (raters are unidentified), e.g., there are 30 objects each with a unique id
  • Each row has the predicted rating, the actual rating, and the difference between the two, which is an ordinal value, say between 0 and 10

How do I measure the consistency of ratings per object and for the whole dataset? Basically, there is a distribution of "differences" per object. What are the various ways to approach this problem? Thank you.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.