I have developed a machine learning model to predict a quantitative output for medical diagnosis (low bone density). I want to convert the model output to a binary outcome and compare it to the gold-standard. The validation set (used to for model selection) and the test set are of similar distributions. I have done ROC analysis on both sets. I am wondering what is the most appropriate set with which to select a threshold for future use in the real world. On one hand, the model was selected based on validation set performance (which could be a source of bias). On the other hand, if using the test set to select a threshold, is this no longer considered to be a true test set since we have changed the way the system operates based on its performance on this set?

Update: There are actually 6 different models (one for each body part to be considered). Dataset sizes vary by model. However, the prevalence of disease in this dataset is not representative of the target population (it is higher due to selection criteria). Validation and Test sets have been stratified to match target population prevalence. For the largest body part set the sizes are as follows:

Training: >10,000 (prevalence is not representative of target) Validation: 622 (stratified to match target) Test: 622 (stratified to match target)

Cost of false positive: 167 dollars, patient goes for extra doctor visit and additional test (not considering patient anxiety from positive result)

Cost of false negative: Currently all of these patient's are missed by the current standard of care. Having the disease would put someone at a 10-20% 10 year risk of fracture which can cost on average $8,000. Number need to treat for therapy ranges between 20-80.

  • 1
    $\begingroup$ Why do you want or need to "convert the model output to a binary outcome"? See this thread for a related example; your model presumably provides an estimate of bone density that would be combined with other clinical factors "in the real world." Also see the problems with making cutoffs in logistic regression, where cutoffs for "classification" often have hidden assumptions. $\endgroup$
    – EdM
    Commented Oct 22, 2021 at 20:26
  • $\begingroup$ I acknowledge the pitfalls of converting the continuous output to a binary one. The short answer for doing this is to fit into existing clinical workflows so that physicians know what to do with the information. The model will be used as a screening test to a diagnostic test and so selecting a threshold is important to determine who should go for the subsequent test. While the right answer is to incorporate other risk factors and not perform the subsequent test all together, it will be some time before evidence is built to do this. $\endgroup$
    – Cicce19
    Commented Oct 22, 2021 at 20:43
  • $\begingroup$ How many cases in your training, validation, and test sets? What are the relative costs of making false-positive and false-negative binary classifications? Please provide that information by editing the question, as comments are easy to overlook, are in a small type size, and can get deleted. $\endgroup$
    – EdM
    Commented Oct 22, 2021 at 21:04
  • $\begingroup$ updated. Not sure why costs of FP and FN are required to determine if test or validation set ROC analysis is most appropriate. I agree they are useful in determining what parameter should be optimized in selection of threshold. Because of the higher prevalence in the dataset, the validation and test sets have been engineered to match the target population. Thus, I believe a bootstrapping approach (if this is what you're thinking) would be challenging. $\endgroup$
    – Cicce19
    Commented Oct 22, 2021 at 21:20
  • 2
    $\begingroup$ @Cicce19, Frank Harrell is a statistician at Vanderbilt and a member of Cross Validated. He has a blog where he addresses this idea of advising physicians. I forget which of these discusses it, but it’s in one of these links. fharrell.com/post/class-damage fharrell.com/post/classification $\endgroup$
    – Dave
    Commented Oct 22, 2021 at 22:20

1 Answer 1


The brief answer is: even if you really, absolutely, positively need to set a cutoff despite the warnings in the links provided by Dave in a comment, don't use either your validation or your test set as the basis. Combine the model with your estimated net costs to do so, if you believe that you have a good model.

As you present the problem, there even seem to be two different cutoffs that you would need to estimate. One is the cutoff for recommending the extra doctor visit and test. The other is for comparison to the "gold standard" in the field. Those aren't necessarily the same, and it's not clear that the second cutoff is needed at all.

The cost-based cutoff for recommending the extra doctor visit and test is pretty straightforward, if you have a correct probability model for needing them and good cost estimates. See this answer, for example. You choose a cutoff at a probability that matches the relative costs. A first pass cutoff with your estimates of 167 dollars for a false-positive and 8000 for a false negative would be a probability of needing the extra visit/test of $\frac{167}{167+8000}= 0.02$; that is, even if the probability of needing the extra visit/test is only 2%, you should recommend it. (This does not, as you note, include other costs/benefits to the patients, or to the provider or insurer.)

Now: how well calibrated is your model around that 2% probability cutoff, when your choice of model was based on a validation set of only 622 individuals? If we are talking about osteoporosis with about 10% prevalence in the target-matched validation group of 622, then your choice of model is based on a validation set having only about 60 individuals with the condition. That seems awfully risky.

That's a problem with your particular train/validation/test-split choice. In your case, the small validation and test sets limit the power that you have to evaluate true model performance. See Frank Harrell's blog post on the dangers of train/test splits, and our previous discussion here. You also should review Harrell's post on classification versus prediction, linked by Dave in a comment. In particular, see his discussion about the sub-sampling of controls that you evidently did to build your training set. That can lead to more problems than it solves.

In terms of comparison against the "gold standard," presumably the FRAX tool, it's not clear why you need a cutoff at all. That tool estimates 10-year probability of fracture, so the critical comparison would be of FRAX 10-year probabilities against the probabilities provided by your model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.