I need to compare the diagnosis of two methods vs a gold standard (all is paired data). The results are categorical variables classified as positive, negative and inconclusive.
How do you deal with the inconclusive category? In a first attempt, I ignored it and computed the sensitivity and specificity with only true positive and true negatives, but that does not seem correct to me as does not give relevant information to choose the best method.
How do you deal with these cases?
The clearest answer I know of comes from "Diagnostic accuracy studies: how to report and analyse inconclusive test results" by Shinkins et al., which the STARD 2015 (Standards for Reporting of Diagnostic Accuracy Studies) guidelines cite as the rationale for their own advice on the matter. The Shinkins paper is short and I recommend reading it in full, but here are some key bits:
Citing the Shinkins paper, the STARD 2015 explanation and elaboration document states:
"Ignoring indeterminate test results can produce biased estimates of accuracy, if these results do not occur at random. Clinical practice may guide the decision on how to handle indeterminate results. There are multiple ways for handling indeterminate test results in the analysis when estimating accuracy and expressing test performance. They can be ignored altogether, be reported but not accounted for or handled as a separate test result category. Handling these results as a separate category may be useful when indeterminate results occur more often, for example, in those without the target condition than in those with the target condition. It is also possible to reclassify all such results: as false positives or false negatives, depending on the reference standard result (‘worst-case scenario’), or as true positives and true negatives (‘best-case scenario’)." (p. 8-9). P. 9., item 16 has more to say about the treatment of invalid inconclusive results (e.g. missing data).
The STARD 2015 explanation and elaboration is only 14 pages (not counting the refs), worth reading in full. Also, since you mention you are comparing the diagnosis of two methods vs. a gold standard, you may find it useful to read through the Guidance on how to use QUADAS-C, a tool to assess risk of bias in comparative diagnostic accuracy studies. QUADAS-C is still in development but there is a public version. This is likely to become a frequently-used standard by which comparative diagnosis studies will be judged in the future.
To expand on @Gabriel's excellent answer into a different direction:
How do you deal with the inconclusive category?
I usually add the invalid and inconclusive categories to the confusion table:
| reference/ | Test/prediction ->
V Gold std. | invalid | negative | uncertain/ | positive
| | | inconclusive |
-------------|---------|----------|--------------|----------
positive | | | |
negative | | | |
The first important information here are:
In a second step, compute figures of merit like sensitivity and specificity and also the percentages of invalid and inconclusive results.
The basic "plain text" definitions of various figures of merit such as sensitivity, specificity, predictive values and so on can immediately be used with such an enhanced confusion maxtrix as well (and also for multi-class and one-class systems and for systems that do not have a closed-world constraint, i.e. where one case can belong to multiple classes*).
As an example, sensitivity is the fraction of cases correctly tested/predicted to be positive among all cases positive according to the gold standard/reference, so: $$ sens = \frac{\#~true~positive}{\#~all~cases~positive~by~gold~standard/reference}$$
I ignored it [inconclusive] and computed the sensitivity and specificity with only true positive and true negatives, but that does not seem correct to me
The often-cited formulation $sens = \frac{TP}{TP+FN}$ is only a derived result for the special case of both reference/gold standard have truly binary (pos/neg) outcome. As you can see, it does not apply here and would overestimate sensitivity (same for specificity) since you'd miss the invalid and inconclusive cases that are positive according to gold standard and should go into the denominator. But the more general definition above works correctly, and sensitivity calculated according to the general definition is a meaningful figure of merit for your test.
Of course, you'll need to compare and judge several figures of merit in order to properly compare gold standard and test. As a minimum, you should consider sensitivity, specificity, predictive values, fraction of invalid test results, fraction of inconclusive test results.
With the latter two you may find that the available data does not allow a good comparison: from your description it looks as if for the gold standard you have only negative and positive, but no invalid or inconclusive outcomes.
You'll need to think why this is: what happened to invalid or inconclusive outcomes of the gold standard? Do they really never happen?
For invalid results you may be able to argue that they occur randomly (at least in some cases that is possible), but inconclusive results can usually not be expected to occur randomly, at least not if inconclusive is what happens at the border between negative and positive.
Inconclusive ("uncertain") test results happen often when some metric response is cut into categories, here negative, inconclusive, positive.
It may be that the inconclusive results do genuinely only occur with your test, and the gold standard is not affected by it. This can be the case if the underlying property is truely binary, your test uses a metric surrogate whereas the gold standard uses either a truly binary surrogate or measures the underlying property directly.
In my experience, this is rare, though.
What I see more often is that cases that result in invalid or inconclusive outcomes in the reference/gold standard are excluded. For a method comparison, this can unfortunately introduce unacceptable bias, and we may not even know the direction of the bias. Let me give examples:
Silently exluding cases that were "difficult" (inconclusive) for the gold standard biases the comparison against the test: we then know only how often the test has difficulties in arriving at a conclusion for cases that were "easy" (conclusive) for the gold standard. But we cannot compare inconclusive test results to inconclusive gold standard results, so the comparison is unfair for the test.
Even worse, we cannot use these results to estimate real-life performance of the test. Since "inconclusive gold standard" may be (hopefully is) positively correlated to "inconclusive test result" (via "difficult/borderline case), such an experimental design for the comparison may underestimate how often the test is inconclusive in reality. Or, to put this into different words, by excluding all difficult (borderline) cases, you may have constructed an artificially easy problem**. In itself, that is OK during early method development. But you cannot conclude real-life performance of your test from such data.
Of course, these difficulties may be negligible if the gold standard is rarely inconclusive. But that statement would require careful justification.
* Side note: the figures of merit can even be extended to deal with situations where the gold standard is either uncertain or fuzzy, e.g. stating that a case is at the borderline between classes.
What happens then is you'll get a possible range for the figures of merit that is in accordance with the gold standard/reference and test result. We describe this in C. Beleites et al.: Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to the Grading of Astrocytoma Tissues, Chemometrics and Intelligent Laboratory Systems, 122 (2013), 12 - 22.
AAM on arXiv
** if you need a citation for this, have a look at our paper C. Beleites et al.: Raman spectroscopic grading of astrocytoma tissues: using soft reference information, Anal. Bioanal. Chem., 400 (2011), 2801 - 2816.
Authors' Accepted Manuscript, incl. supplementary information
The paper discusses this mainly from the perspective of method development/classifier training, but also points out the consequences for testing/verification/validation.
