I have data with a binary outcome and I am doing logit model selection using AIC and BIC. I have already withheld 30% of the data as a holdout sample (testing subset) and used the remainder (training subset) to do model selection.
In order to calculate accuracy, sensitivity, specificity, PPV, NPV and all of those parameters, I need a threshold. I plan on using Youden's index to maximize the difference between my test and the random chance line.
However, in generating the ROC curve, do I use the training data to generate the curve and choose a threshold, and then apply this threshold to the predicted values for the testing data? Or do I generate the ROC curve using the testing dataset and pick the threshold from that?
In the former case, I am generating an ROC curve based on data that were used to make the model, which seems like it would give me a falsely-high AUC (since the models are fit to that particular data), and the threshold chosen won't necessarily be the best threshold. But in the latter case, I am generating an ROC curve based on the testing data, data that the models have not seen, and picking an optimal threshold from this data. This seems a little cheaty as well, since I can pick the threshold that gives me the highest sensitivity/specificity for the testing subset, but this might not be the case for generalizing to the intended population.
TL;DR: do I pick my threshold based on an ROC curve generated with the model testing or model training data?
Thanks.