I am trying to calculate inter-rater reliability. Previous researchers in this area have used intraclass correlation. SPSS has options for two-way random, mixed random and one-way random models. SPSS help says choose the right one based on whether the 'people effects are random' and 'item effects are random', can anybody please explain to me what these terms mean?
Also I wanted to know how to structure my data file - I think I need to put each raters' rating of each item in a separate column (rather than each rater on a separate row), is this right?
Thanks to the help I have received, I have visited the websites, which were useful. I have decided that I need to use the mixed model, and I have transposed my files so that each rater is a column.
The websites also mentioned some assumptions, one of which is additivity, tested with Tukey's test of non-additivity. Apparently, this tests the hypothesis that there is no multiplicative interaction between the cases and the items. I think maybe I have too many variables in my calculation, because the examples others use have a lot less, e.g. each rater making only one rating, as described below, my raters have made many more ratings than that.
- What does the additivity assumption mean?
- What should be done when the test of non-additivity is statistically significant?
Inter-rater reliability with multiple objects and properties
I will explain in more detail what I have done. I have five raters and they have rated facial and vocal expressions of participants watching different emotional films. Raters make 18 ratings per film, and there are 16 films, so there is a total of 288 variables per rater, per participant rated. Each rater has rated the same 4 participants each, and I have placed these participants into different files. I thought about making seperate files for each of the 16 films, but that would be so many files.
- How should I calculate ICC and organise my files in this specific situation?
- Should I calculate a mean of individual ICCs or perhaps select a random sample?