Does anyone have recommendations on how to evaluate test retest reliability with 95%CI (individuals took the same survey 1 week apart) on data that is dichotomous (yes = 1; no = 0) where the data is highly skewed (Test: 165 = yes; 1 no) and (Retest: 166 = yes 0 = no).

When running an ICC the values act as if the test retest reliability is poor (clearly as the data is skewed and not continuous or normally distributed therefore not meant for analysis using ICC). However, similar issues arise when trying other measures of association (e.g. phi coefficient, kendall's tau-b which of course provide the same value of 0.06) indicating low agreement (although seemingly high).

Has anyone had this issue or knows of a better method of reporting? I was considering using McNemar's but essentially you get a p value of 1.000 and unsure what other data you can report that would be equivalent of a measure of "reliability" similar to that of ICC. I am not using kappa as it equally affected by skewed distribution and am not evaluating inter-rater reliability but instead test retest of a single person taking a survey twice.

Any suggestions are vastly appreciated. I am doing analysis in STATA 13