I need to compare the diagnosis of two methods vs a gold standard (all is paired data). The results are categorical variables classified as positive, negative and inconclusive.

How do you deal with the inconclusive category? In a first attempt, I ignored it and computed the sensitivity and specificity with only true positive and true negatives, but that does not seem correct to me as does not give relevant information to choose the best method.

How do you deal with these cases?


The clearest answer I know of comes from "Diagnostic accuracy studies: how to report and analyse inconclusive test results" by Shinkins et al., which the STARD 2015 (Standards for Reporting of Diagnostic Accuracy Studies) guidelines cite as the rationale for their own advice on the matter. The Shinkins paper is short and I recommend reading it in full, but here are some key bits:

  • They first distinguish between "valid" and "invalid" inconclusive results. "Invalid" inconclusive results are those which are 'uninterpretable' in the sense that they “do not meet the minimum criteria constituting an adequate test” or the actual test result is missing. "Valid" inconclusive results are those where an adequate test result has been obtained, but the result is not clearly positive or negative. Invalid inconclusive results should be reported and their reasons described, but they should not be lumped together with valid conclusive results.
  • "Inconclusive results can be difficult to analyse given that many statistics used to summarise the accuracy of diagnostic tests require the test results to be split into two groups. There is no single “optimal” approach to analysing inconclusive results; diagnostic accuracy should always be analysed in line with how the test will be used in clinical practice." They identify three main approaches:
    • Excluding valid inconclusive results completely: "there are few instances where this can be justified, and this approach can lead to overstated summary statistics and promotion of suboptimal test strategies."
    • "Another method is to exclude valid inconclusive results from accuracy statistics such as sensitivity and specificity, but report an additional statistic that takes into account the presence of inconclusive results." Such statistics include "test yield" and "effectiveness", and they give examples of the calculation of each. Upsides are that this is more transparent. Downsides are that "the risk of simply providing an additional statistic to account for inconclusive results is that readers might struggle to interpret such unfamiliar statistics and interpret only the more popular accuracy measures, such as sensitivity and specificity. Furthermore, these additional statistics are not typically included in meta-analyses, where usually only the sensitivity and specificity are analysed."
    • Finally, one can "group [valid inconclusive results] with either the positive or negative results, depending on how these patients would be treated in the clinical context", and they give an example. Because deciding whether to group inconclusive results with the positives or the negatives can make a big difference to measures such as sensitivity and specificity, they highlight the benefits of also reporting a secondary analysis where the decision was made differently. And of course, being completely transparent about which group the inconclusive results were grouped with in the primary analysis and why.

Citing the Shinkins paper, the STARD 2015 explanation and elaboration document states:

"Ignoring indeterminate test results can produce biased estimates of accuracy, if these results do not occur at random. Clinical practice may guide the decision on how to handle indeterminate results. There are multiple ways for handling indeterminate test results in the analysis when estimating accuracy and expressing test performance. They can be ignored altogether, be reported but not accounted for or handled as a separate test result category. Handling these results as a separate category may be useful when indeterminate results occur more often, for example, in those without the target condition than in those with the target condition. It is also possible to reclassify all such results: as false positives or false negatives, depending on the reference standard result (‘worst-case scenario’), or as true positives and true negatives (‘best-case scenario’)." (p. 8-9). P. 9., item 16 has more to say about the treatment of invalid inconclusive results (e.g. missing data).

The STARD 2015 explanation and elaboration is only 14 pages (not counting the refs), worth reading in full. Also, since you mention you are comparing the diagnosis of two methods vs. a gold standard, you may find it useful to read through the Guidance on how to use QUADAS-C, a tool to assess risk of bias in comparative diagnostic accuracy studies. QUADAS-C is still in development but there is a public version. This is likely to become a frequently-used standard by which comparative diagnosis studies will be judged in the future.

  • 1
    $\begingroup$ "if these [indeterminate] results do not occur at random" in my experience, the valid inconclusive results rarely occur at random (some mechanisms for invalid data occur at random, but in general I wouldn't even rely on invalid data occuring at random). In particular, borderline cases are usually even supposed to give inconclusive results. $\endgroup$ – cbeleites unhappy with SX Jun 23 '20 at 9:30

To expand on @Gabriel's excellent answer into a different direction:

How do you deal with the inconclusive category?

I usually add the invalid and inconclusive categories to the confusion table:

| reference/ | Test/prediction -> 
V Gold std.  | invalid | negative | uncertain/   | positive
             |         |          | inconclusive |             
positive     |         |          |              |             
negative     |         |          |              |             

The first important information here are:

  • Is there indication that invalid and/or inconclusive results are unevenly distributed over the gold standard outcomes?
    If so, in particularly for the invalid test results, you'll need to find out what happens.

In a second step, compute figures of merit like sensitivity and specificity and also the percentages of invalid and inconclusive results.

The basic "plain text" definitions of various figures of merit such as sensitivity, specificity, predictive values and so on can immediately be used with such an enhanced confusion maxtrix as well (and also for multi-class and one-class systems and for systems that do not have a closed-world constraint, i.e. where one case can belong to multiple classes*).

As an example, sensitivity is the fraction of cases correctly tested/predicted to be positive among all cases positive according to the gold standard/reference, so: $$ sens = \frac{\#~true~positive}{\#~all~cases~positive~by~gold~standard/reference}$$

I ignored it [inconclusive] and computed the sensitivity and specificity with only true positive and true negatives, but that does not seem correct to me

The often-cited formulation $sens = \frac{TP}{TP+FN}$ is only a derived result for the special case of both reference/gold standard have truly binary (pos/neg) outcome. As you can see, it does not apply here and would overestimate sensitivity (same for specificity) since you'd miss the invalid and inconclusive cases that are positive according to gold standard and should go into the denominator. But the more general definition above works correctly, and sensitivity calculated according to the general definition is a meaningful figure of merit for your test.

Of course, you'll need to compare and judge several figures of merit in order to properly compare gold standard and test. As a minimum, you should consider sensitivity, specificity, predictive values, fraction of invalid test results, fraction of inconclusive test results.

With the latter two you may find that the available data does not allow a good comparison: from your description it looks as if for the gold standard you have only negative and positive, but no invalid or inconclusive outcomes.

You'll need to think why this is: what happened to invalid or inconclusive outcomes of the gold standard? Do they really never happen?

For invalid results you may be able to argue that they occur randomly (at least in some cases that is possible), but inconclusive results can usually not be expected to occur randomly, at least not if inconclusive is what happens at the border between negative and positive.

The reasons behind inconclusive test results

Inconclusive ("uncertain") test results happen often when some metric response is cut into categories, here negative, inconclusive, positive.

It may be that the inconclusive results do genuinely only occur with your test, and the gold standard is not affected by it. This can be the case if the underlying property is truely binary, your test uses a metric surrogate whereas the gold standard uses either a truly binary surrogate or measures the underlying property directly.
In my experience, this is rare, though.

What I see more often is that cases that result in invalid or inconclusive outcomes in the reference/gold standard are excluded. For a method comparison, this can unfortunately introduce unacceptable bias, and we may not even know the direction of the bias. Let me give examples:

Silently exluding cases that were "difficult" (inconclusive) for the gold standard biases the comparison against the test: we then know only how often the test has difficulties in arriving at a conclusion for cases that were "easy" (conclusive) for the gold standard. But we cannot compare inconclusive test results to inconclusive gold standard results, so the comparison is unfair for the test.

Even worse, we cannot use these results to estimate real-life performance of the test. Since "inconclusive gold standard" may be (hopefully is) positively correlated to "inconclusive test result" (via "difficult/borderline case), such an experimental design for the comparison may underestimate how often the test is inconclusive in reality. Or, to put this into different words, by excluding all difficult (borderline) cases, you may have constructed an artificially easy problem**. In itself, that is OK during early method development. But you cannot conclude real-life performance of your test from such data.

Of course, these difficulties may be negligible if the gold standard is rarely inconclusive. But that statement would require careful justification.

* Side note: the figures of merit can even be extended to deal with situations where the gold standard is either uncertain or fuzzy, e.g. stating that a case is at the borderline between classes.
What happens then is you'll get a possible range for the figures of merit that is in accordance with the gold standard/reference and test result. We describe this in C. Beleites et al.: Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to the Grading of Astrocytoma Tissues, Chemometrics and Intelligent Laboratory Systems, 122 (2013), 12 - 22.
AAM on arXiv

** if you need a citation for this, have a look at our paper C. Beleites et al.: Raman spectroscopic grading of astrocytoma tissues: using soft reference information, Anal. Bioanal. Chem., 400 (2011), 2801 - 2816.
Authors' Accepted Manuscript, incl. supplementary information

The paper discusses this mainly from the perspective of method development/classifier training, but also points out the consequences for testing/verification/validation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.