Skip to main content
2 of 2
edited tags
chl
  • 54.3k
  • 23
  • 227
  • 388

Measuring inter-subject agreement in user ratings

In my system I have users rating objects (e.g. films) from 1 to 5 stars. The mean rating clearly tells something about the overall opinion on an object, but I'd like to use a more precise measure of variability. For example, if the stars are set randomly, if half of the users give 1 and the other half is 5, the average is still 3, so it doesn't tell me the agreement among users. I'd like a measure that tells me the inter-subject agreement, ranging from 1=all users agree, 0=random ratings.

What would be a good and standard measure for this aspect of a set of ratings?

Mulone
  • 295
  • 2
  • 11