I am trying to compare the reliability of two different assessment methods (method 1 and method 2). Each method assesses task performance for a single cohort of students (N = 50).

For method 1, student performance is rated by 4 judges (who vary across students). The assessment method assigns students a numeric score.

For method 2, student performance is rated by 5 judges (judges are fixed and consistent across students). The assessment method assigns a numeric score based solely on narrative comments.

I now want to compare the reliability of the two assessment methods.

I have calculated ICCs, but I am not sure how to best compare whether they are equivalent. I have 95% CIs, and can assess degree of overlap. Is there something more "sophisticate"?

  • $\begingroup$ Not clear what you are asking. $\endgroup$ – Michael R. Chernick Jan 27 '17 at 20:47
  • $\begingroup$ I'm not sure I understand. Are there 50 students in total being judged on two tasks or are there 50 students in two different groups doing different tasks? $\endgroup$ – Vilgot Huhn Jan 27 '17 at 23:45

Formal tests for comparing ICC, mostly based on F-tests, were proposed, e.g. McGraw, K.O., and Wong, S.P., Forming Inferences about Some Intraclass Correlation Coefficients, Psychological Methods, 1(1): 30–46 (1996). However, bootstrap resampling on the difference between the two ICCs is also an option: you can construct a 90% or 95% confidence interval using the percentile method (or bias-correction) and check the lower (or upper) confidence limit to decide on whether one ICC is larger than the other. More generally, working with confidence intervals should be enough. This is what we often use to estimate the number of subkects needed for a study, for instance.

E.g., assuming a theoretical reliability of 0.8, we want to determine the number of subjects needed to be 95% confident that the observed reliability estimated from a random sample is > 0.7; this amounts to define a 95% confidence interval centered around 0.1 (0.8 - 0.7), hence a standard error of 0.05. With 5 observations per subject, we would need 36 subjects in total.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.