1
$\begingroup$

I have some clinical diagnostic data from three sites that I have been looking at performance against a reference standard using the McNemar test.

How would I set up the analysis to investigate if there are any differences between the three sites?

My data (in R) look like:

myData <- read.table(header=TRUE, text='
Site ReferenceStandard NewDiagnostic Count
A R R 47
A S R 2
A R S 0
A S S 387
B R R 3
B S R 1
B R S 7
B S S 161
C R R 13
C S R 0
C R S 0
C S S 108')

ct <- xtabs(Count ~ ReferenceStandard + NewDiagnostic + Site, data=myData)

ftable(ct)
                                Site   A   B   C
ReferenceStandard NewDiagnostic                 
R                 R                   47   3  13
                  S                    0   7   0
S                 R                    2   1   0
                  S                  387 161 108

Do I use the Cochran–Mantel–Haenszel?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

McNemar's test is for paired data but CMH is not. What kinds of differences are you looking for? If you're interested in heterogeneity of the prevalence of the outcome, fit a logistic model with site effects for the reference values only (this will give you independent data). If you're interested in heterogeneity in the agreements, just mutually compare the 95% CIs for the matched ORs or fit a conditional logistic regression model.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I was afraid that it would not work (or be as powerful) because I have paired data. If you look at the site level data, there appear to some big differences, especially Site B, but it is due to very small numbers. I edited the question to push some data... $\endgroup$
    – user918967
    Aug 15, 2016 at 22:44
  • $\begingroup$ My main goal is to see if there are statistically significant 'site differences' in sensitivity/specificity to help understand a site was performing the test poorly. $\endgroup$
    – user918967
    Aug 15, 2016 at 22:52
  • $\begingroup$ @user918967 you can generate 95% confidence intervals for $c$-statistics or ROC curves and compare them for each site. You shouldn't be using a Mantel Haenszel to find interrater agreement, very miscalibrated measures can be considered highly accurate. Use a kappa statistic instead. $\endgroup$
    – AdamO
    Aug 15, 2016 at 23:00

Not the answer you're looking for? Browse other questions tagged or ask your own question.