My question concerns the assumption of additivity for intraclass correlation. I shall first explain what I have done and then end with my questions.
I want to calculate inter-rater reliability using intra-class correlation so I can report an overall coefficient (as done in previous similar research), and perhaps replace a rater if their judgements correspond poorly to the other raters. I have five raters and they have each rated video recordings of facial and vocal expressions of the same (randomly sampled) 4 participants in an experiment where the participants watched different emotional films.
Raters make 18 ratings per film. These ratings are Likert-type (generally ranging from 1-6 but for some measures 1-4) of the intensity of different 6 facial emotional expressions (anger, fear etc), the intensity of facial expression overall, and the number (frequency ratings) and intensity (Likert ratings) of positive and negative words and sounds, and level of overall vocal expressiveness.
There are 16 films, so there is a total of 288 variables per rater, per participant rated. I have organised my data into four files, one per participant being rated, with each rater as a column and the 288 variables as rows. As I am calculating inter-rater reliability, I am interested in the similarity of the raters overall, and not any other (e.g. film) effects.
I have calculated the ICC using the mixed model because all judges rate all targets, which are a random sample (as per http://faculty.chass.ncsu.edu/garson/PA765/reliab.htm#rater)
Questions: The assumption of additivity states that each item should be linearly related to the total score. However I don’t think that the concept of a total score really applies, although I may be wrong. Tukey’s test of non-additivity tests the null hypothesis that there is no multiplicative interaction between cases and items.
- Could somebody please explain this to me in simple terms?
I found a significant Tukey’s test value, so I tried removing the overall facial and vocal ratings for each film, as I thought this perhaps violated the requirement that each item contributes to the total score. However Tukey’s test remained significant. So just as a little experiment, I removed 282 variables, leaving me with ratings of the 6 possible facial emotions for a single film. Tukey’s test was still significant!
Is Tukey’s test of non-additivity relevant to my problem?
If yes, what should I do about it being significant?