From one perspective, the concept of retest reliability breaks down in the context of discrete data that's this heavily skewed. Retest reliability is a matter of how well between-subject differences are preserved from time 1 to time 2, and in this case, there are basically no between-subject differences to be preserved. If the base rate for your test really is something like 1 in 150 or 1 in 200 people getting a "no", then getting a good estimate of the between-subject differences at baseline, let alone how well they are preserved from time 1 to time 2, would require a much larger sample; I'd want at least, say, 10 subjects with each label at baseline, in which case you'd need something on the order of 1,500 subjects.

Another way to look at retest reliability is as a measure of well scores obtained at time 1 can predict scores for the same subjects on the same tests at time 2, or how close the time-1 scores are to being equal to time-2 scores. This is the notion of retest reliability I used in Arfer and Luhmann (2016). From this perspective, your test is extremely reliable because nearly all subjects got the same score at both timepoints. A simple way to quantify this is with percent agreement: 99.4% of subjects got the same score at both timepoints.

Arfer, K. B., & Luhmann, C. C. (2016). Time-preference tests fail to predict self-control behavior. Retrieved from http://arfer.net/projects/rickrack/paper

From one perspective, the concept of retest reliability breaks down in the context of discrete data that's this heavily skewed. Retest reliability is a matter of how well between-subject differences are preserved from time 1 to time 2, and in this case, there are basically no between-subject differences to be preserved. If the base rate for your test really is something like 1 in 150 or 1 in 200 people getting a "no", then getting a good estimate of the between-subject differences at baseline, let alone how well they are preserved from time 1 to time 2, would require a much larger sample; I'd want at least, say, 10 subjects with each label at baseline, in which case you'd need something on the order of 1,500 subjects.

Another way to look at retest reliability is as a measure of how well scores obtained at time 1 can predict scores for the same subjects on the same tests at time 2, or how close the time-1 scores are to being equal to time-2 scores. This is the notion of retest reliability I used in Arfer and Luhmann (2017). From this perspective, your test is extremely reliable because nearly all subjects got the same score at both timepoints. A simple way to quantify this is with percent agreement: 99.4% of subjects got the same score at both timepoints.

Arfer, K. B., & Luhmann, C. C. (2017). Time-preference tests fail to predict behavior related to self-control. Frontiers in Psychology, 8(150). doi:10.3389/fpsyg.2017.00150. Retrieved from http://arfer.net/projects/rickrack/paper