4
$\begingroup$

For a given dataset, a common way to compare 2 classifiers is to compare their average validation accuracies using cross-validation.

Is it valid to replace the accuracy with other classification metrics that I care more about? For example, say I care about the sensitivity (recall) at a given specificity level (say 0.99). Is it still valid to compare A and B by computing the average sensitivities using cross-validation? (for each fold, train the model, plot the ROC curve, get the recall for specificity=0.99)

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ No; this is at the opposite end of the spectrum from decision making. See fharrell.com/post/mlconfusion $\endgroup$
    – Frank Harrell
    Jun 4, 2022 at 16:26
  • $\begingroup$ It's not a "no" because, in the end, it's a metric for model evaluation just like many others. The validity of the proposed metric is not in question. It's a no if someone thinks it's a good way to do so. $\endgroup$
    – gunes
    Jun 5, 2022 at 7:14

1 Answer 1

Reset to default
3
$\begingroup$

Yes, you can choose any metric of interest while you're validating your model. This can be a specific metric of your choice, like sensitivity value at a specific specificity as you proposed, if you think that best suits your need; or area under ROC, precision/recall curve, F1-score etc. The space of possible choices is not limited to accuracy.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.