I am interested in estimating the following:
- test-retest reliability for ordinal items;
- test-retest reliability for interval-level scale scores consisting of the items; and
- the optimal sample size for (2)
(1) I believe weighted kappa is the best option for test-retest reliability for ordinal items. Correct?
(2) I'm finding conflicting advice: Pearson r or some flavor of ICC.
(3) Shoukri et al. (2004)* report the following about methods for finding the optimal sample size for estimating ICC:
Giraudeau and Mary and Bonett argued that the approach of hypothesis testing may not be appropriate while planning a reliability study. This is because one has to specify both the values of ρ0 and ρ1, which may in practice be a difficult task. An alternative approach is to instead focus on the width [w] of the CI for ρ.
I believe this table applies to test-retest as well as inter-rater, but I'm not certain.
If so, I'm concluding that one would need 52 participants and n=2 time points to estimate ρ=0.80 with a 95% CI width of 0.20 (so 0.7-0.9, I assume). Does this seem accurate?
*Shoukri, M. M., Asyali, M. H., & Donner, A. (2004). Sample size requirements for the design of reliability study: review and new results. Statistical Methods in Medical Research, 13(4), 251-271.