# Intraclass correlation coefficient interpretation

I'm having a look at the intraclass correlation coefficient in SPSS.

Data: 17 participants rated two lists of 9 & 7 items from 0 to 5 (0 being unimportant and 5 being very important).

All participants rated all the items and the participants are a sample of a large population.

The following output has been produced in SPSS.

I am struggling to find anything online which deals with interpreting this, nor does any book interpret this in the level of detail I need.

There is surprisingly little information/examples of the interpretation online on this, the literature is about choosing an intraclass correlation coefficient and not interpreting it.

One problem I foresee here is the F test. The data is only 17 responses, which do not follow the normality assumption.

I am struggling to find anything online which deals with interpreting this

The output you present is from SPSS Reliability Analysis procedure. Here you had some variables (items) which are raters or judges for you, and 17 subjects or objects which were rated. Your focus was to assess inter-rater aggreeement by means of intraclass correlation coefficient.

In the 1st example you tested p=7 raters, and in the 2nd you tested p=9.

More importantly, your two outputs differ in the respect how the raters are considered. In the 1st example, the raters are a fixed factor, which means they are the population of raters for you: you infer about only these specific raters. In the 2nd example, the raters are a random factor, which means they are a random sample of raters for you, while you want infer about the population of all possible raters which those 9 pretend to represent.

The 17 subjects that were rated constitute a random sample of population of subjects. And, since each rater rated all 17 subjects, both models are complete two-way (two-factor) models, one is fixed+random=mixed model, the other is random+random=random model.

Also, in both instances you requested to assess the consistency between raters, that is, how well their ratings correlate, - rather than to assess the absolute agreement between them - how much identical their scores are. With measuring consistency, Average measures ICC (see the tables) are identical to Cronbach's alpha. Average measures ICC tells you how reliably the/a group of p raters agree. Single measures ICC tells you how reliable is for you to use just one rater. Because, if you know the agreement is high you might choose to inquire from just one rater for that sort of task.

If you tested the same number of the same raters (and the same subjects) under both models you'd see that the estimates in the table are the same under both models. However, as I've said, the interpretation differs in that you can generalize the conclusion about the agreement onto the whole population of raters only with two-way random model. You can see also a footnote saying that the mixed model assumes there is no rater-subject interaction; to put clearer, it means that the raters lack individual partialities to subjects' characteristics not relevant to the rated task (e.g. to hair colour of an examenee).

SPSS Reliability Analysis procedure assumes additivity of scores (which logically implies interval or dichotomous but not ordinal level of data) and bivariate normality between items/raters. However, F test is quite robust.

• This relates to two questions from an administered questionnaire. Say the second one is the correct model, how do you go about interpreting the numbers within the table. Commented Jul 18, 2013 at 18:36
• I don't understand your comment. Essentially, all models are wrong, but some are useful. Commented Jul 18, 2013 at 18:43
• Consider the second picture above. How would you deal with interpreting the numbers within it. I want "to assess the consistency between raters, that is, how well their ratings correlate", so I agree with that. Commented Jul 18, 2013 at 18:46

You might want to read the article by LeBreton and Senter (2007). It's a fairly accessible overview of how to interpret ICC and related indicators of inter-rater agreement.

LeBreton, J. M., & Senter, J. L. (2007). Answers to 20 questions about interrater reliability and interrater agreement. Organizational Research Methods.

Let me provide a response for the first situation that you analysed because the second situation essentially parallels it, except that you have two more items in the second situation and you chose a different model (more about that below). In providing this response, in some places I have a different interpretation from the extended explanation that has been provided elsewhere in these posts.

As I understand it, you had 17 raters (participants), each of whom provided a rating on 5-point scales to seven different items AND you are wanting to see whether there is much agreement between the 17 raters in how they rated those 7 items. I think that, in order to do this (which is surely a pretty unusual situation; usually there are not as many as 17 raters involved in assessing something), you should have selected ABSOLUTE (not consistent) measures in SPSS, and, if your participants are the only raters of interest in this situation (I assume they are, and that you are not wanting to generalize your results to other participants / raters) you should indeed have chosen Model 3 (i.e., 2-way mixed, NOT Model 2 as you did in your second setup), which is the FIRST model offered in SPSS. So, in essence, you have made a basic mistake in selecting the kind of ICC that provides a consistency solution SPSS. (Sorry to give you the bad news.)

Next, when you choose an ICC from the output you should choose the ICC from the row titled "Single measures" (i.e., .133) because each of your participants made a single rating for each of the 7 items (and I assume you entered 17 scores into the ICC analysis for each item). If you had averaged all of your 17 participants' ratings on each item BEFORE entering the data into the ICC analysis, it would be appropriate for you to report the ICC that pertains to the Averaged measures (.519). But, from your description, you didn't average the ratings that were made by your participants.

If you had chosen Absolute rather than Consistency for your first analysis, an ICC as low as .133 would indicate that your 17 participants / raters exhibited EXTREMELY little agreement among themselves in terms of how they rated the 7 items.

An article in 2016 by Trevethan in the journal Health Services and Outcomes Research Methodology provides the background for this answer as well as a lot of other information concerning the selection and interpretation of ICCs.

Finally, the small number of items (7 in the first situation) might create some problems statistically. I am sorry, but I am not able to provide advice about that. Maybe it's OK in your situation, but it might be advisable to consult a friendly statistician.

References

Trevethan, R. (2016). "Intraclass correlation coefficients: Clearing the air, extending some cautions, and making some requests." Health Services and Outcomes Research Methodology. DOI 10.1007/s10742-016-0156-6. (Online publication available until volume, issue, and page numbers have been assigned.)

• Could you add a full citation for the Trevethan (2016) article? Commented Aug 24, 2016 at 16:31
• Sure, here's the full reference for the Trevethan (2016) article. The article has just been published and I didn't have the full reference because the volume, issue, and page numbers have not yet been assigned. Here is as much as I can provide: Trevethan, R. (2016). Intraclass correlation coefficients: Clearing the air, extending some cautions, and making some requests. Health Services and Outcomes Methodology. The DOI has been assigned, but I'm not sure it's yet operating as of 26 August 2016. It is 10.1007/s10742-016-0156-6. Commented Aug 26, 2016 at 9:47
• I've added that into the article (feel free to edit to change the formatting) - usually best not to leave things in the comments, partly because people don't always read them and partly because comments are best treated as transient (can't be guaranteed they will be around forever!) Commented Aug 26, 2016 at 10:58
• Thanks for this. Appreciated. In my haste I made a mistake, however. The journal title is Health Services and Outcomes Research Methodology. If I can't alter it in the post, please feel free to do it on my behalf. Cheers, RT. Commented Aug 26, 2016 at 16:57

I have traced the answer in new Stata 13 documentation on ICC.

The question remains on whether the F test in this case can be used given the data does not follow the assumptions of normal distribution.