What is more important in a test set sensitivity/specificity or ROC AUC

Question

I am working on a diagnostic test (I'm not a statistician) I used a logit model for regression with the help of another student who has since graduated.

I separated my data into a Training and Test Set. In the training set I have an AUC of 0.94 and a sensitivity/specificity of .87/.92.

In the test set I have an AUC of 0.88 (which seems acceptable) but the sensitivity/specificity is .43/.92 if I use the same threshold as in the training set.

Is it an error to change the threshold? and also how can I describe my data in terms of its ability in diagnosis. What could cause this?

Would it be reasonable to say that "the test seems to have a reproducible specificity but many cases may be missed" or does this mean that the validation completely failed.

Training set ROC

Test Set ROC

Answer 1

If you used k-fold cross validation (CV) you would be in a better position to evaluate these performance metrics. Splitting data into training and testing one time (and then determining performance: sens, spec, AUC) is inefficient and does not provide an optimal measure of performance (see Ron Kohavi's paper on CV). I like to repeat 10-fold CV a total of ten times, whereby each time I shuffle the order of the objects so that the folds contain different objects each time -- this is called a $repartition$, and in the end the process is called "ten 10-fold CV."

Sounds like you used CV for feature selection. If you trained using all the objects and then tested with a sub-sample, the classifier learned information from objects that are in the test set, which is biased. CV prevents that from happening. For example, use 10-fold CV. Permute (randomly shuffle) the objects and divide uniformly into 10 folds (partitions). Train with folds 2-10, and test with the 1st fold. Then train with the 1st and 3rd-10th folds, and test with the 2nd fold. Continue until you train with folds 1-9 and test with the 10th fold.

You can average sens, spec, and AUC over the 10 folds, or just keep padding (appending) ones to the confusion matrix throughout testing of the ten folds, and then determine sens, spec, AUC with the single confusion matrix. To understand the latter, for each tested object, a one is added to confusion matrix element in the row representing the true class, and the column representing the predicted class. When done, classification accuracy is the sum of diagonal elements divided by sum of all matrix elements, because diagonal elements represent objects whose true class and predicted class are the same. Sens, spec are just variations on this theme.

Answer 2

Generally speaking, there is ample literature on characteristics of diagnostic tests (eg the Joanna Briggs Institute Guidelines or the Cochrane Methods Section on Diagnostic Test Accuracy Studies) detailing which statistics should be computed in a diagnostic test accuracy study.

Briefly, you should first identify true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN):

Then, you can compute:

• prevalence, defined as (TP+FN)/(TP+FN+TN+FP);
• sensitivity,defined as TP/(TP+FN);
• specificity, defined as TN/(TN+FP);
• diagnostic odds ratio (DOR);
• positive predictive value (+PV);
• negative predictive value (-PV);
• positive likelihood ratio (+LR), defined as sensitivity/(1-specificity);
• negative likelihood ratio (-LR), defined as (1-sensitivity)/specificity;
• area under the curve (AUC) of the receiver operating curve (ROC).

The most accurate and robust statistics are +LR, -LR, and AUC, as they are less dependent on disease prevalence. In addition, diagnostic tests may have prognostic features, and you could also envision randomization to a diagnostic test.

Let's now focus on your questions:

Is it an error to change the threshold?

It could be risky, amounting to data massaging, unless you have sound clinical reasons to do that. Remember that the threshold you are proposing to readers is one only and should be stable for subsequent clinical application.

How can I describe my data in terms of its ability in diagnosis?

That's difficult to say without looking at the actual curves. It might be you have few cases and a few misdiagnoses are very impactful.

What could cause this?

As above, if your sample is small with few or many diseased cases even few misdiagnoses can be impactful on sensitivity and specificity.

Would it be reasonable to say that "the test seems to have a reproducible specificity but many cases may be missed" or does this mean that the validation completely failed?

I would also compute DOR, +LR, and -LR. If they are consistent, you can simply state that in your sample sensitivity is not the appropriately stable and precise statistics to describe the diagnostic accuracy of your test.