# McNemar's test probably does not work, but what could?

I have a dataset of people who had back-to-back procedures and I have the whether or not polyps (precancerous lesions) removed during each procedure.

Subject  Pass.1   Pass.2   Overall
1        1        1        1
2        0        1        1
3        1        0        1
4        0        0        0
...


(where 1 = a polyp was found)

I want to know if compared with the first procedure, there was an increased detection rate with back−to−back colonoscopy. McNemar's test seems like a possibility, but I don't think it would apply since the detection rate can only increase (i.e. the "No/Yes" cell in the contingency table will always be 0). This is the table I have for setting up the McNemar test:

                      After Both Procedures
After First Procedure     No Yes
No  64  13
Yes  0  118


Would McNemar's test apply? If not, is it possible to do the analysis with a different test or at all?

• Is it really true that it is impossible for a polyp to be found on the first procedure, but no polyps were found on the second procedure? If so, what is the substantive scientific question here? Is it theoretically coherent? – gung May 1 '18 at 16:52
• It is possible, but I'm comparing doing one procedure versus doing both (the Pass.1 and Overall columns). Since any polyps found in the first procedure are removed, the second procedure will only yield additional polyps. Did that answer your question? – radishhorse May 1 '18 at 17:08
• Does that really make sense as a question? Can the rate go down by establishing a detection in the first pass as a false positive? If not, the existence of a single detection (anywhere, ever) on a subsequent pass unequivocally proves that the rate is higher. (That isn't a statistical conclusion, but a logical one.) – gung May 1 '18 at 17:19
• It's completely possible that the question does not make sense in that detection can only increase. I only asked because I found a paper that did a similar setup and did the comparison (link). They found that "Compared with the first colonoscopy, there was an increased diagnosis rate by back−to−back colonoscopy for patients with polyps (42% [95% CI 37±48] vs. 66% [60±71]; P < 0.0001)" I think they used the McNemar's test but it is possible that they did that incorrectly. – radishhorse May 1 '18 at 17:31

You have a 2x2 table, where you are interested in comparing the marginal proportions. That is appropriate for McNemar's test.

A different question is whether this makes any sense to do. It is rather unfortunate that a mindless and mechanical application of statistics has taken over much of science, such that conclusions can only be considered 'scientific' or valid if they are accompanied by p-values. The standard version of McNemar's test makes sense here if subsequent procedures can establish that a detection in the first pass was a false positive. Then you could have non-$0$ values in the lower left "yes no" cell. Otherwise, the existence of a single detection (anywhere, ever) on a subsequent pass unequivocally proves that the rate is higher. That isn't a statistical conclusion, but a logical one.
\begin{align} &P \supset Q \\ &P \\ &\hline \\[-50pt] &\therefore Q \end{align} Modus ponens does not require a statistical test.

Setting aside the idiocy of testing, it is certainly possible to estimate the improvement in detection rates. Moreover, you could get a 95% confidence interval around the improvement. Further, if you wanted to test if the improvement were different (above / below) from some meaningful threshold, that could be done. That might make sense as part of a process of assessing if the improvement in the rates outweighed the cost of additional procedures (either literally, or in terms of the risk of side effects, etc.).

• You're welcome, @radishhorse. – gung May 1 '18 at 20:33

You are looking for Chochran's Q test, which extends the logic of McNemar's test for proportion difference in paired data to an omnibus test for proportion difference in blocked data (akin to how the repeated measures ANOVA extends the logic of the paired t test for mean difference of normally distributed data to an omnibus test for mean difference in blocked data). Post hoc pairwise tests can either be conducted using McNemer's test, or Cochran's Q test, as the latter corresponds to the former on two groups.

Software Implementations

Cochran's Q test has been implemented for Stata, in the cochranq package (which you can access within Stata by typing net describe cochranq, from(https://alexisdinno.com/stata). This package includes multiple comparisons adjustments for post hoc pairwise comparisons, and options to estimate the effect size using either the Serlin, Carr and Marascuillo, or the Berry, Johnston and Mielke methods.

Cochran's Q test has been implemented for R in the cochran.q program in the nonpar package on CRAN (which you can install within R by typing install.packages(c("nonpar"))). This package does not provide multiple comparisons adjustments or effect size measures.

• Does this still work if the "treatments" are dependent on each other? I wanted to compare "before second procedure" and "after second procedure" -- so if before = 1, then after = 1. Like I said with @gung above, it's entirely possible that the comparison itself doesn't make sense, but I did see it done in a paper that made me think it was possible (though they possibly misused the McNemar's test). – radishhorse May 1 '18 at 17:55
• It's not ideal (loses some power), but with effect size and interpretation of the direction of differences I think it's reasonable. – Alexis May 1 '18 at 19:29
• Thanks for your help! Do you agree with @gung that it doesn't make sense to ask the question at all since detection can only increase? – radishhorse May 1 '18 at 20:35
• @radishhorse Perhaps gung and I agree with that. – Alexis May 1 '18 at 21:01