# Picking a model threshold based on Validation set or Test set

I have developed a machine learning model to predict a quantitative output for medical diagnosis (low bone density). I want to convert the model output to a binary outcome and compare it to the gold-standard. The validation set (used to for model selection) and the test set are of similar distributions. I have done ROC analysis on both sets. I am wondering what is the most appropriate set with which to select a threshold for future use in the real world. On one hand, the model was selected based on validation set performance (which could be a source of bias). On the other hand, if using the test set to select a threshold, is this no longer considered to be a true test set since we have changed the way the system operates based on its performance on this set?

Update: There are actually 6 different models (one for each body part to be considered). Dataset sizes vary by model. However, the prevalence of disease in this dataset is not representative of the target population (it is higher due to selection criteria). Validation and Test sets have been stratified to match target population prevalence. For the largest body part set the sizes are as follows:

Training: >10,000 (prevalence is not representative of target) Validation: 622 (stratified to match target) Test: 622 (stratified to match target)

Cost of false positive: 167 dollars, patient goes for extra doctor visit and additional test (not considering patient anxiety from positive result)

Cost of false negative: Currently all of these patient's are missed by the current standard of care. Having the disease would put someone at a 10-20% 10 year risk of fracture which can cost on average \$8,000. Number need to treat for therapy ranges between 20-80.

• Why do you want or need to "convert the model output to a binary outcome"? See this thread for a related example; your model presumably provides an estimate of bone density that would be combined with other clinical factors "in the real world." Also see the problems with making cutoffs in logistic regression, where cutoffs for "classification" often have hidden assumptions.
– EdM
Oct 22 '21 at 20:26
• I acknowledge the pitfalls of converting the continuous output to a binary one. The short answer for doing this is to fit into existing clinical workflows so that physicians know what to do with the information. The model will be used as a screening test to a diagnostic test and so selecting a threshold is important to determine who should go for the subsequent test. While the right answer is to incorporate other risk factors and not perform the subsequent test all together, it will be some time before evidence is built to do this. Oct 22 '21 at 20:43
• How many cases in your training, validation, and test sets? What are the relative costs of making false-positive and false-negative binary classifications? Please provide that information by editing the question, as comments are easy to overlook, are in a small type size, and can get deleted.
– EdM
Oct 22 '21 at 21:04
• updated. Not sure why costs of FP and FN are required to determine if test or validation set ROC analysis is most appropriate. I agree they are useful in determining what parameter should be optimized in selection of threshold. Because of the higher prevalence in the dataset, the validation and test sets have been engineered to match the target population. Thus, I believe a bootstrapping approach (if this is what you're thinking) would be challenging. Oct 22 '21 at 21:20
• @Cicce19, Frank Harrell is a statistician at Vanderbilt and a member of Cross Validated. He has a blog where he addresses this idea of advising physicians. I forget which of these discusses it, but it’s in one of these links. fharrell.com/post/class-damage fharrell.com/post/classification
– Dave
Oct 22 '21 at 22:20

The cost-based cutoff for recommending the extra doctor visit and test is pretty straightforward, if you have a correct probability model for needing them and good cost estimates. See this answer, for example. You choose a cutoff at a probability that matches the relative costs. A first pass cutoff with your estimates of 167 dollars for a false-positive and 8000 for a false negative would be a probability of needing the extra visit/test of $$\frac{167}{167+8000}= 0.02$$; that is, even if the probability of needing the extra visit/test is only 2%, you should recommend it. (This does not, as you note, include other costs/benefits to the patients, or to the provider or insurer.)