3
$\begingroup$

I am currently writing my thesis about inter-rater reliability of a diagnostic tool between raters. I want to use the standard error of measurement with the formula:

SEM of rater 1 and rater 2 = SD * $\sqrt{1-ICC}$ where SD represents standard deviation and ICC represents the reliability of rater 1 and rater 2.

However, I cannot find what SD is necessary... Do I use the pooled SD of the 2 raters?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ There are two measures of ICC. One is for the average score, one is individual score. In R, these are ICC1 and ICC2 (I forget which package, sorry). In Stata, they are both given as well when you use the loneway function. $\endgroup$
    – Jeremy Miles
    Commented Nov 25, 2015 at 17:49

2 Answers 2

Reset to default
1
$\begingroup$

As Jeremy pointed out, there are multiple versions of the ICC, that reflect distinct ways of accounting for raters or items variance in overall variance. There's a nice summary of the use of Kappa and ICC indices for rater reliability in Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial, by Kevin A. Hallgren, and I discussed the different versions of ICC in a related post. Briefly, you have to decide whether your raters are considered as sampled from a larger pool of potential raters or as a fixed set of raters. In the first case, this means using a random effect model, while in the second raters will be treated as fixed effects. Likewise, items may be treated as either fixed or random units. Usually, we use a two-way random effects model (both raters and items are treated as random effects) to estimate relative agreement (or the ICC(2,k) version for absolute reliability, if you care about systematic error between raters). The SEM can be calculated from the square root of the mean square error from a one-way ANOVA, or from the sample standard deviation, as suggested in the other reply.

Note that the choice of SD doesn't matter as much as that of the ICC, since they can differ a lot depending on your sample size and the inherent variation of the ratings. Here are two examples of results obtained from the same dataset using the ICC2 (top) and ICC3 (bottom) approaches:

enter image description here

enter image description here

In R, there are many packages available, including psych (see ICC) and irr. I provide some examples of use of the former in my other answer, and I have some examples of the use of irr in a separate blog post on my site. Using Stata, the icc command provides all three ANOVA models and associated estimation for the ICC, as shown above.

Improve this answer
$\endgroup$
0
$\begingroup$

I think it should be either

  1. the SD from a large number of measurements from a single sample, or

  2. sqrt[SStotal/(n-1)] from all subjects.

The first is the correct one, however often unrealistic to achieve. The latter is the alternative/ an estimation.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ I think this answer would be improved by making it clear whether the "either" means that, so far as you know, both would work, or is simply an acknowledgement that you don't know which? $\endgroup$
    – Silverfish
    Commented Apr 14, 2016 at 21:05
  • 1
    $\begingroup$ It's best to edit your answer to reflect posts like this - people might not always read the comments, or the comments might even be deleted at some point in the future. (They are "second class citizens" on Stack Exchange sites.) $\endgroup$
    – Silverfish
    Commented Apr 15, 2016 at 8:30
  • 2
    $\begingroup$ Could someone explain what each of the individual components from the equation "sqrt[SStotal/(n-1)]" stand for? Thank you very much. $\endgroup$
    – user112317
    Commented Apr 15, 2016 at 12:53
  • 1
    $\begingroup$ SS = sum of squares ; n= sample size; sqrt = squareroot $\endgroup$
    – rbru
    Commented Apr 23, 2016 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.