I am interested in estimating the following:

  1. test-retest reliability for ordinal items;
  2. test-retest reliability for interval-level scale scores consisting of the items; and
  3. the optimal sample size for (2)

(1) I believe weighted kappa is the best option for test-retest reliability for ordinal items. Correct?

(2) I'm finding conflicting advice: Pearson r or some flavor of ICC.

(3) Shoukri et al. (2004)* report the following about methods for finding the optimal sample size for estimating ICC:

Giraudeau and Mary and Bonett argued that the approach of hypothesis testing may not be appropriate while planning a reliability study. This is because one has to specify both the values of ρ0 and ρ1, which may in practice be a difficult task. An alternative approach is to instead focus on the width [w] of the CI for ρ.

I believe this table applies to test-retest as well as inter-rater, but I'm not certain.

enter image description here

If so, I'm concluding that one would need 52 participants and n=2 time points to estimate ρ=0.80 with a 95% CI width of 0.20 (so 0.7-0.9, I assume). Does this seem accurate?

*Shoukri, M. M., Asyali, M. H., & Donner, A. (2004). Sample size requirements for the design of reliability study: review and new results. Statistical Methods in Medical Research, 13(4), 251-271.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.