4
$\begingroup$

I have data with a binary outcome and I am doing logit model selection using AIC and BIC. I have already withheld 30% of the data as a holdout sample (testing subset) and used the remainder (training subset) to do model selection.

In order to calculate accuracy, sensitivity, specificity, PPV, NPV and all of those parameters, I need a threshold. I plan on using Youden's index to maximize the difference between my test and the random chance line.

However, in generating the ROC curve, do I use the training data to generate the curve and choose a threshold, and then apply this threshold to the predicted values for the testing data? Or do I generate the ROC curve using the testing dataset and pick the threshold from that?

In the former case, I am generating an ROC curve based on data that were used to make the model, which seems like it would give me a falsely-high AUC (since the models are fit to that particular data), and the threshold chosen won't necessarily be the best threshold. But in the latter case, I am generating an ROC curve based on the testing data, data that the models have not seen, and picking an optimal threshold from this data. This seems a little cheaty as well, since I can pick the threshold that gives me the highest sensitivity/specificity for the testing subset, but this might not be the case for generalizing to the intended population.

TL;DR: do I pick my threshold based on an ROC curve generated with the model testing or model training data?

Thanks.

Improve this question
$\endgroup$
1
  • $\begingroup$ What are you actually using this model for? $\endgroup$
    – shadowtalker
    May 11, 2016 at 5:10

2 Answers 2

Reset to default
5
$\begingroup$

Generate the ROC curve and choose the threshold within the training data, but then report accuracy, sensitivity, etc. when using this threshold to make predictions in the test data.

AUC is not a great metric, but if you want it (and you don't want it to be optimistically biased), generate another ROC curve for the test data and report the AUC for that.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you, this approach makes sense. I understand that AUC is not great, but is there an alternative that I could report instead? $\endgroup$
    – user17325
    May 12, 2016 at 22:33
  • $\begingroup$ An alternative to AUC for what? What were you hoping to accomplish with AUC? $\endgroup$
    – Kodiologist
    May 13, 2016 at 0:57
  • $\begingroup$ Could you explain a bit about the logic behind using "the threshold within the training data, but then report accuracy, sensitivity, etc. when using this threshold to make predictions in the test data"? From my totally inexperienced view, I would think that using the testing threshold would be better since it would give a better picture of how the model would generalize. $\endgroup$
    – lampShadesDrifter
    Mar 6, 2018 at 20:38
  • 2
    $\begingroup$ @lampShadesDrifter Using the testing threshold may optimistically bias your estimates of accuracy etc., due to ovefitting. If you use the training threshold, by contrast, then the model will be properly penalized for overfitting. $\endgroup$
    – Kodiologist
    Mar 6, 2018 at 23:40
  • 2
    $\begingroup$ @lampShadesDrifter Overfitting can happen in the training set, just as easily as it would in the test set if you fit the model with the test set. The reason you make the train–test split is so that if the model overfits, you'll see this correctly as a decrease in accuracy (when the model is asked to make predictions on the test set after having been fit on the training set), instead of a false increase in apparent accuracy. $\endgroup$
    – Kodiologist
    Mar 7, 2018 at 2:15
4
$\begingroup$

There are so many issues with your approach that it difficult to know where to start. Some issues to consider:

  • You need at least n=20,000 for split-sample validation to be stable and reliable
  • sens, spec, ppv, npv, proportional "classified" "correctly" are all highly inefficient, arbitrary, discontinuous improper accuracy scores that are optimized by choosing the wrong variables and giving them the wrong coefficients
  • Model selection is often a bad idea, as opposed to just forming a well thought-out fully specified model
  • The use of thresholds is completely at odds with optimum decision making
  • The logistic model is intended to be used to estimate probabilities
  • ROC curves should play no role in estimation or decision making
  • The utility/cost/loss function cannot come from the data themselves
Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you for your input, Dr. Harrell, I appreciate your time and help. I am starting with ~65,000 cases, so this should be OK. Regarding sensitivity/specificity/etc, I am not sure I follow your concern. I am not doing model selection based on these values, so I do not understand how they would affect the variables and coefficients in the model. I was planning to calculate these terms as estimates of the performance of the model. To calculate these, I must convert predicted probabilities to decisions that the model would make, to know if the model was right or wrong. Should I use 0.5? $\endgroup$
    – user17325
    May 12, 2016 at 22:30
  • $\begingroup$ Models do not make decisions. Models make estimates. Thresholds not informed by utility functions are inconsistent with optimum Bayes decisions. Sens and spec condition on the future to estimate probabilities about the past, which is unhelpful. And why are you doing model selection? $\endgroup$
    – Frank Harrell
    May 12, 2016 at 22:34
  • $\begingroup$ Your work in this field is undeniable. However, this response reads a bit like an arraignment hearing ;) I am interested in following up on some of your points. What is the recommended reference? $\endgroup$
    – jeffalltogether
    Sep 9, 2019 at 20:10
  • $\begingroup$ fharrell.com/tags/classification and fharrell.com/post/backwards-probs $\endgroup$
    – Frank Harrell
    Sep 11, 2019 at 11:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.