I have $n$ individuals, and for each individual, I have two measurements using two devices (device X and device Y). I know the ground truth for the correct measurement, and I can classify each measurement as accurate or inaccurate. Thus, for each individual I effectively have a boolean value that indicates whether device X was correct or not (say $x_i$) and a boolean value that indicates whether device Y was correct or not (say $y_i$).
Is there a good statistical test to use to compare the accuracy rate of the two devices?
In particular, suppose I notice that device X's accuracy rate appears to be higher than device Y's accuracy rate, based upon the $n$ observations (i.e., $(x_1+\dots+x_n)/n > (y_1+\dots+y_n)/n$, where $x_i,y_i = 1$ means it was correct and $0$ means it was incorrect). Now I'd like to test whether the difference in observed accuracy rate is statistically significant. Can I compute a $p$-value for the null hypothesis that their underlying accuracy rate is actually the same?
Should I use the Wilcoxon signed-rank test? A paired Student's t-test? Some sort of paired Welch t-test (does such a thing even exist)? None of those seems like an obvious fit to me: I know the data isn't normally distributed (it presumably has a Bernoulli distribution), so a t-test isn't perfect (on the other hand I've read that in practice the t-test is fairly robust to deviations from normality so maybe it is OK?); and I can't tell whether a Wilcoxon signed-rank test takes into account the prior knowledge that the data is Bernoulli distributed. Anyway, what would be the most appropriate methodology?
