7
$\begingroup$

For example, if my table is:

                True Value (gold standard)
                   Positive  | Negative |
         |        |          |          |
Test     | Pos    |    A     |     B    |   
Result   |        |          |          |
         | Neg    |    C     |     D    |
         |        |          |          |
$\endgroup$
  • 5
    $\begingroup$ It's not clear to me that there can be a useful answer to this question. The receiver operating characteristic (ROC) curve represents the range of tradeoffs between true-positive and false-positive classifications as one alters the threshold for making that choice from the model. A contingency table represents the classification results at a particular choice of that threshold. One might be able to calculate something like an area (as one proposed answer here does), but it's not clear that would truly represent the area under the ROC curve for the full model. $\endgroup$ – EdM Oct 17 '18 at 17:31
  • 1
    $\begingroup$ If your predictor is dichotomous, and there is therefore only one threshold, I think the AUC still provides (some) useful information. $\endgroup$ – Jeremy Miles Oct 17 '18 at 20:18
18
$\begingroup$

In the general case: you can't

The ROC curve shows how sensitivity and specificity varies at every possible threshold. A contingency table has been calculated at a single threshold and information about other thresholds has been lost. Therefore you can't calculate the ROC curve from this summarized data.

But my classifier is binary, so I have one single threshold

Binary classifiers aren't really binary. Even though they may expose only a final binary decision, all the classifiers I know rely on some quantitative estimate under the hood.

  • A binary decision tree? Try to build a regression tree.
  • A classifier SVM? Do a support vector regression.
  • Logistic regression? Get access to the raw probabilities.
  • Neural network? Use the numeric output of the last layer instead.

This will give you more freedom to choose the optimal threshold to get to the best possible classification for your needs.

But I really want to

You really shouldn't. ROC curves with few thresholds significantly underestimate the true area under the curve (1). A ROC curve with a single point is a worst-case scenario, and any comparison with a continuous classifier will be inaccurate and misleading.

Just give me the answer!

Ok, ok, you win. With a single point we can consider the AUC as the sum of two triangles T and U:

A ROC curve with a single (SP, SE) pair and two triangles

We can get their areas based on the contingency table (A, B, C and D as you defined):

$$ \begin{align*} T = \frac{1 \times SE}{2} &= \frac{SE}{2} = \frac{A}{2(A + C)} \\ U = \frac{SP \times 1}{2} &= \frac{SP}{2} = \frac{D}{2(B + D)} \end{align*} $$

Getting the AUC: $$ \begin{align*} AUC &= T + U \\ &= \frac{A}{2(A + C)} + \frac{D}{2(B + D)} \\ &= \frac{SE + SP}{2} \end{align*} $$

To conclude

You can technically calculate a ROC AUC for a binary classifier from the confusion matrix. But just in case I wasn't clear, let me repeat one last time: DON'T DO IT!

References

(1) DeLong ER, DeLong DM, Clarke-Pearson DL: Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach. Biometrics 1988,44:837-845. https://www.jstor.org/stable/2531595

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

When I claim all of them are negative, then sensitivity (y) = 0, 1 - specificity (x) = 0. If I claim the positive/negative according to test results, then y =A/(A+C), x=B/(B+D). When I say all of them are Positive, then y = 1 and x = 1.

Based on three points with coordinate (0,0) (A/(A+C), B/(B+D)) (1,1), (in (y,x) order), it is easy to calculate the area under the curve by using the formula for area of triangle.

Final result: Area = $\frac {AB+2AD+2CD}{(A+C)(B+D)}$ ? Need to be verified.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.