0
$\begingroup$

Given two binary diagnostic tests, T1 and T2, I want to compare the Sensitivity and Specificity of the two tests when given to a sample of subjects. Each subject will receive both tests, and there is a gold standard source of "truth" that the diagnostics are attempting to predict. With this, I have a paired comparison of Sensitivity or Specificity and McNemar's Test seems the most appropriate.

It has been suggested that to compare Sensitivities between the two tests, I need to find the score cutoff values for each test that provide the same Specificity value, and then I can compare the Sensitivities with McNemar's tests. The opposite has also been suggested - to compare Specificities, the Sensitivities must be the same.

Given that this is a validation of two different tests, each test has a specific score cutoff value that was determined during test development. It seems that the Sensitivities and Specificities observed at those predetermined cutoff values should be the compared rather than adjust the cutoffs to get either Sensitivity or Specificity the same and then comparing the other.

Am I interpreting this correctly? It seems to me that the observed Sensitivities and Specificities at the predetermined cutoff values for each test is how this comparison should be made. Or am I misunderstanding?

$\endgroup$

1 Answer 1

1
$\begingroup$

First, let me say that I share your surprise at the suggestions which were given. I would be very curious at hearing the rationale for such suggestions (changing cutoff value to align either Sensitivity or Specificity: the whole point of the cutoff is that it is chosen as the "optimal tradeoff" between Sensitivity and Specificity).
Based on your comment, I assume you are interested in comparing the 2 tests "as is", that is, as sold by their manufacturers, as offered on the market. Therefore, ignore this suggestion.
Just a point which you did not mention, but I am sure you are aware off. To assess Sensitivity and Specificity, you need a 3rd test, a gold standard, or other clinical evidence which establishes the true nature of the condition. Otherwise you can not even compute Sensitivity and Specificity.
Another point, which may not be as obvious, is that the McNemar test does not compare Sensitivities/Specificities; it tests for marginal homogeneity, i.e. it compares how often the 2 tests disagrees with each other, not how often they are "correct". So I would not recommend using a McNemar test.
Let's look for a little bit at a slightly different, but relevant scenario. Let's say that we are still looking at 2 different diagnostic tests, but they are in fact exactly the same test, but at 2 different cutoffs. How would one compare their Sensitivity/Specificity? Simply by plotting them on a ROC plot (see wiki), and picking the cutoff which corresponds to the point closest to the top left corner. Ideally one also plots confidence intervals around these 2 points (after all, they are just estimates), and then indeed confidence intervals around the distances. One could also decide to assign different cost functions to the false positives and false negatives (this depends on the test, the condition, the use case, etc.); this just amounts to weighing the 2 axis differently, and then still minimizing the distance from the top left corner.
Well, if you intend to compare the 2 tests in question, you should use a ROC plot, estimate the Sensitivities and Specificities of the 2 tests, and place the 2 points on a ROC plot. You can then see which is closest to the top left corner. Note though that these points are estimates, and have uncertainty (the CI's), so while one may be closer to the top left corner, that may not be statistically significant. You also will have paired data (same subjects under both tests, and under the "gold standard"). So you can do further analysis; look at how often they both were wrong (Flase positive or false negative for both), vs. only 1 of the tests was wrong, vs. they were both right (TP or TN for both); that will tell you how redundant the 2 tests may be (they agree -right or wrong- for most subjects), or how complimentary they could be (one catches the errors of the other). This may be even more interesting than finding out that they have very similar Sensitivities/Specificities.

$\endgroup$
1
  • $\begingroup$ Thank you for this excellent response! There is a gold standard test that serves as "truth" - I'll add that to the original post. After doing some additional research, the ROC plot and a comparison of AUCs with DeLong's Test. This seems a more valid solution that picking an arbitrary value for Sensitivity/Specificity and then testing Specificity/Sensitivity at that value. As you mention, the test should be done in the as-is state from the manufacturer. I can see the ROC plot giving a nice indicator of performance across a range of cutoffs. $\endgroup$
    – KirkD_CO
    Commented Sep 6 at 17:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.