Picking a model threshold based on Validation set or Test set

Question

I have developed a machine learning model to predict a quantitative output for medical diagnosis (low bone density). I want to convert the model output to a binary outcome and compare it to the gold-standard. The validation set (used to for model selection) and the test set are of similar distributions. I have done ROC analysis on both sets. I am wondering what is the most appropriate set with which to select a threshold for future use in the real world. On one hand, the model was selected based on validation set performance (which could be a source of bias). On the other hand, if using the test set to select a threshold, is this no longer considered to be a true test set since we have changed the way the system operates based on its performance on this set?

Update: There are actually 6 different models (one for each body part to be considered). Dataset sizes vary by model. However, the prevalence of disease in this dataset is not representative of the target population (it is higher due to selection criteria). Validation and Test sets have been stratified to match target population prevalence. For the largest body part set the sizes are as follows:

Training: >10,000 (prevalence is not representative of target) Validation: 622 (stratified to match target) Test: 622 (stratified to match target)

Cost of false positive: 167 dollars, patient goes for extra doctor visit and additional test (not considering patient anxiety from positive result)

Cost of false negative: Currently all of these patient's are missed by the current standard of care. Having the disease would put someone at a 10-20% 10 year risk of fracture which can cost on average $8,000. Number need to treat for therapy ranges between 20-80.

Answer 1

The brief answer is: even if you really, absolutely, positively need to set a cutoff despite the warnings in the links provided by Dave in a comment, don't use either your validation or your test set as the basis. Combine the model with your estimated net costs to do so, if you believe that you have a good model.

As you present the problem, there even seem to be two different cutoffs that you would need to estimate. One is the cutoff for recommending the extra doctor visit and test. The other is for comparison to the "gold standard" in the field. Those aren't necessarily the same, and it's not clear that the second cutoff is needed at all.

The cost-based cutoff for recommending the extra doctor visit and test is pretty straightforward, if you have a correct probability model for needing them and good cost estimates. See this answer, for example. You choose a cutoff at a probability that matches the relative costs. A first pass cutoff with your estimates of 167 dollars for a false-positive and 8000 for a false negative would be a probability of needing the extra visit/test of $\frac{167}{167+8000}= 0.02$; that is, even if the probability of needing the extra visit/test is only 2%, you should recommend it. (This does not, as you note, include other costs/benefits to the patients, or to the provider or insurer.)

Now: how well calibrated is your model around that 2% probability cutoff, when your choice of model was based on a validation set of only 622 individuals? If we are talking about osteoporosis with about 10% prevalence in the target-matched validation group of 622, then your choice of model is based on a validation set having only about 60 individuals with the condition. That seems awfully risky.

That's a problem with your particular train/validation/test-split choice. In your case, the small validation and test sets limit the power that you have to evaluate true model performance. See Frank Harrell's blog post on the dangers of train/test splits, and our previous discussion here. You also should review Harrell's post on classification versus prediction, linked by Dave in a comment. In particular, see his discussion about the sub-sampling of controls that you evidently did to build your training set. That can lead to more problems than it solves.

In terms of comparison against the "gold standard," presumably the FRAX tool, it's not clear why you need a cutoff at all. That tool estimates 10-year probability of fracture, so the critical comparison would be of FRAX 10-year probabilities against the probabilities provided by your model.