Test Retest of highly skewed dichotomous variables

Question

Does anyone have recommendations on how to evaluate test retest reliability with 95% CI (individuals took the same survey 1 week apart) on data that is dichotomous (yes = 1; no = 0) where the data is highly skewed (Test: 165 = yes; 1 no) and (Retest: 166 = yes 0 = no).

When running an ICC the values act as if the test retest reliability is poor (clearly as the data is skewed and not continuous or normally distributed therefore not meant for analysis using ICC). However, similar issues arise when trying other measures of association (e.g. phi coefficient, Kendall's tau-b which of course provide the same value of 0.06) indicating low agreement (although seemingly high).

Has anyone had this issue or knows a better method of reporting? I was considering using McNemar's but essentially you get a p value of 1.000 and unsure what other data you can report that would be equivalent of a measure of "reliability" similar to that of ICC. I am not using kappa as it is equally affected by skewed distributions and am not evaluating inter-rater reliability but instead test retest of a single person taking a survey twice.

Any suggestions are vastly appreciated. I am doing analysis in Stata 13.

Answer 1

From one perspective, the concept of retest reliability breaks down in the context of discrete data that's this heavily skewed. Retest reliability is a matter of how well between-subject differences are preserved from time 1 to time 2, and in this case, there are basically no between-subject differences to be preserved. If the base rate for your test really is something like 1 in 150 or 1 in 200 people getting a "no", then getting a good estimate of the between-subject differences at baseline, let alone how well they are preserved from time 1 to time 2, would require a much larger sample; I'd want at least, say, 10 subjects with each label at baseline, in which case you'd need something on the order of 1,500 subjects.

Another way to look at retest reliability is as a measure of how well scores obtained at time 1 can predict scores for the same subjects on the same tests at time 2, or how close the time-1 scores are to being equal to time-2 scores. This is the notion of retest reliability I used in Arfer and Luhmann (2017). From this perspective, your test is extremely reliable because nearly all subjects got the same score at both timepoints. A simple way to quantify this is with percent agreement: 99.4% of subjects got the same score at both timepoints.

Arfer, K. B., & Luhmann, C. C. (2017). Time-preference tests fail to predict behavior related to self-control. Frontiers in Psychology, 8(150). doi:10.3389/fpsyg.2017.00150. Retrieved from http://arfer.net/projects/rickrack/paper