How to calculate inter-rater reliability in extremely large and sparse data sets?

I have 28k raters across 140k items. I am attempting to measure the IRR.

The data in the matrix is extremely sparse, the minimum ratings of a rater is 20 and the maximum is 9254.

Is using Fleiss going to be worthwhile here?

• Would you clarify what an "IRR" is? And when you write "minimum ratings" are you referring to the number of ratings? Nov 1, 2015 at 13:40
• By IRR I mean inter rater reliability, or inter rater agreement. Yes I mean the minimum number of items a rater has rated is 20, and the maximum is 9254. Nov 1, 2015 at 13:49
• Would you elaborate on what the raters are rating as well as the scale in which the ratings are couched? In other words, what information is in these 140k items? Nov 1, 2015 at 13:52
• There is a numerical rating between 0 and 5 in 0.5 increments. The raters (users) are ratings movies. Nov 1, 2015 at 14:13
• So, these are ratings for 140k movies? Why not think in terms of making your analysis more tractable by reducing some of the dimensionality? E.g., collapse the scales into 0,1 You could screen out infrequently rated movies. Movies belong to genres and the individuals doing the ratings are likely to have preferences by genre: do the analysis by genre. Movies have a production year: do it by year or decade. Movies have additional features that could be folded in such as production studio -- e.g., determining if there is greater agreement for movies made by Buena Vista vs Universal, and so on Nov 1, 2015 at 14:27

As long as some movies have been rated by two or more raters, you can assess reliability using Fleiss' kappa. Given the ordinal scaling from 0 to 5, you will want to implement a weighting scheme. Format the data into an $n$-by-$r$ matrix where $n$ is the number of movies (140,000) and $r$ is the number of raters (28,000); each movie gets its own row and each rater gets his/her own column. In each cell of the matrix put that rater's rating of that movie or NaN. Now send that matrix to a function such as mSCOTTPI which will calculate the chance-adjusted agreement using a generalized formula that can handle multiple categories, multiple raters, missing data, and a weighting scheme.