Running an ICC with variable number of raters

Question

I have a dataset where 69 videos are rated by 130 raters, but each rater only rates 10 videos each. I need to calculate some measure of rater reliability, but everything I have attempted won't run because of the missing data.

After some reading, I have found that ICC(1) might be appropriate for this type of data, however, when I go to run this model in SPSS the output says there are not enough variables to calculate (assuming because of listwise deletion to deal with missing data).

Before I go insane working out how to run this, can anyone enlighten me as to how I might actually run the ICC(1) on the data?

I have my data set up as columns for the raters and rows for the videos.

Thanks!

Answer 1

First, make sure your data is set up properly. Second, make sure that each video was rated by two or more raters. Third, make sure you are using the proper ICC formulation. Lastly, make sure you are using the ICC function properly.

1. You should set up your data as a matrix with each rater as a separate column and each row as a separate video. So you should end up with a $69\times130$ matrix. Make sure that your missing cells (i.e., rater-video combinations) are marked as "missing" in whatever program you are using.

2. In most functions, each video needs to be rated by two or more raters. Other videos should be thrown away (or a specialized function that accounts for this needs to be used).

3. I would not recommend ICC(1) for your described purpose. This formulation assumes that raters are not a meaningful source of variance, which they almost certainly are. Instead, you should use ICC(A,1) or ICC(C,1); these are sometimes called two-way mixed models for single-measures and either absolute agreement or consistency. Use ICC(A,1) if you want the raters to use the exact same scores and be fully interchangeable; use ICC(C,1) if you're okay with each rater having their own mean.

4. This will depend on the exact function you are using, but make sure you send the data to the function in the exact way that it expects.

If you use MATLAB, I have made the ICC_A_1 and ICC_C_1 functions available.

If you use SPSS, you can use the RELIABILITY function.

Give it a shot; if you run into problems, feel free to send me the data and I can do it for you (or at least seeing the data will give me more insight into the problem).