I have developed a machine learning model to predict a quantitative output for medical diagnosis (low bone density). I want to convert the model output to a binary outcome and compare it to the gold-standard. The validation set (used to for model selection) and the test set are of similar distributions. I have done ROC analysis on both sets. I am wondering what is the most appropriate set with which to select a threshold for future use in the real world. On one hand, the model was selected based on validation set performance (which could be a source of bias). On the other hand, if using the test set to select a threshold, is this no longer considered to be a true test set since we have changed the way the system operates based on its performance on this set?
Update: There are actually 6 different models (one for each body part to be considered). Dataset sizes vary by model. However, the prevalence of disease in this dataset is not representative of the target population (it is higher due to selection criteria). Validation and Test sets have been stratified to match target population prevalence. For the largest body part set the sizes are as follows:
Training: >10,000 (prevalence is not representative of target) Validation: 622 (stratified to match target) Test: 622 (stratified to match target)
Cost of false positive: 167 dollars, patient goes for extra doctor visit and additional test (not considering patient anxiety from positive result)
Cost of false negative: Currently all of these patient's are missed by the current standard of care. Having the disease would put someone at a 10-20% 10 year risk of fracture which can cost on average $8,000. Number need to treat for therapy ranges between 20-80.