2
$\begingroup$

I am working on a diagnostic test (I'm not a statistician) I used a logit model for regression with the help of another student who has since graduated.

I separated my data into a Training and Test Set. In the training set I have an AUC of 0.94 and a sensitivity/specificity of .87/.92.

In the test set I have an AUC of 0.88 (which seems acceptable) but the sensitivity/specificity is .43/.92 if I use the same threshold as in the training set.

Is it an error to change the threshold? and also how can I describe my data in terms of its ability in diagnosis. What could cause this?

Would it be reasonable to say that "the test seems to have a reproducible specificity but many cases may be missed" or does this mean that the validation completely failed.

Training Training set ROC

Test set Test Set ROC

$\endgroup$
  • 1
    $\begingroup$ What would my final metric be of how well the experiment was working be if I did this. I am hesitant to do cross validation because I already determined which variables to measure by doing cross validation with re-sampling, on the initial set. I then trained the model on the entirety of the initial set. And then applied it to the test set. $\endgroup$ – Christopher Feb 9 '17 at 2:47
  • 1
    $\begingroup$ Can you plot the two ROC curves? $\endgroup$ – not_bonferroni Feb 9 '17 at 14:57
  • 1
    $\begingroup$ Irrespective of how you did this the drop in performance seems quite startling. Can you give us more details like the composition of the diagnostic test, the sample size, the prevalence of disease, and anything else which might help us. $\endgroup$ – mdewey Feb 9 '17 at 15:05
  • $\begingroup$ Thanks so much for your help For the initial set the DOR, is 78.75 the +LR is 14.5 and the -LR is 0.14 there are 48 cases and 48 controls (this was known before hand in both sets) For the test set the DOR is 18.68, the +LR is 11.25 and the -LR is .57 there are 29 cases and 24 controls Are the results on the test set reliable, or does the difference in the two tests suggest that both sets are unreliable. $\endgroup$ – Christopher Feb 9 '17 at 19:28
1
$\begingroup$

If you used k-fold cross validation (CV) you would be in a better position to evaluate these performance metrics. Splitting data into training and testing one time (and then determining performance: sens, spec, AUC) is inefficient and does not provide an optimal measure of performance (see Ron Kohavi's paper on CV). I like to repeat 10-fold CV a total of ten times, whereby each time I shuffle the order of the objects so that the folds contain different objects each time -- this is called a $repartition$, and in the end the process is called "ten 10-fold CV."

Sounds like you used CV for feature selection. If you trained using all the objects and then tested with a sub-sample, the classifier learned information from objects that are in the test set, which is biased. CV prevents that from happening. For example, use 10-fold CV. Permute (randomly shuffle) the objects and divide uniformly into 10 folds (partitions). Train with folds 2-10, and test with the 1st fold. Then train with the 1st and 3rd-10th folds, and test with the 2nd fold. Continue until you train with folds 1-9 and test with the 10th fold.

You can average sens, spec, and AUC over the 10 folds, or just keep padding (appending) ones to the confusion matrix throughout testing of the ten folds, and then determine sens, spec, AUC with the single confusion matrix. To understand the latter, for each tested object, a one is added to confusion matrix element in the row representing the true class, and the column representing the predicted class. When done, classification accuracy is the sum of diagonal elements divided by sum of all matrix elements, because diagonal elements represent objects whose true class and predicted class are the same. Sens, spec are just variations on this theme.

$\endgroup$
1
$\begingroup$

Generally speaking, there is ample literature on characteristics of diagnostic tests (eg the Joanna Briggs Institute Guidelines or the Cochrane Methods Section on Diagnostic Test Accuracy Studies) detailing which statistics should be computed in a diagnostic test accuracy study.

Briefly, you should first identify true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN):

enter image description here

Then, you can compute:

  • prevalence, defined as (TP+FN)/(TP+FN+TN+FP);
  • sensitivity,defined as TP/(TP+FN);
  • specificity, defined as TN/(TN+FP);
  • diagnostic odds ratio (DOR);
  • positive predictive value (+PV);
  • negative predictive value (-PV);
  • positive likelihood ratio (+LR), defined as sensitivity/(1-specificity);
  • negative likelihood ratio (-LR), defined as (1-sensitivity)/specificity;
  • area under the curve (AUC) of the receiver operating curve (ROC).

The most accurate and robust statistics are +LR, -LR, and AUC, as they are less dependent on disease prevalence. In addition, diagnostic tests may have prognostic features, and you could also envision randomization to a diagnostic test.

Let's now focus on your questions:

Is it an error to change the threshold?

It could be risky, amounting to data massaging, unless you have sound clinical reasons to do that. Remember that the threshold you are proposing to readers is one only and should be stable for subsequent clinical application.

How can I describe my data in terms of its ability in diagnosis?

That's difficult to say without looking at the actual curves. It might be you have few cases and a few misdiagnoses are very impactful.

What could cause this?

As above, if your sample is small with few or many diseased cases even few misdiagnoses can be impactful on sensitivity and specificity.

Would it be reasonable to say that "the test seems to have a reproducible specificity but many cases may be missed" or does this mean that the validation completely failed?

I would also compute DOR, +LR, and -LR. If they are consistent, you can simply state that in your sample sensitivity is not the appropriately stable and precise statistics to describe the diagnostic accuracy of your test.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.