1
$\begingroup$

I am working on a classification problem. Several models are produced and all have accuracy, precision and recall metrics on test data. I need to pick the best model among the alternatives. What I can think of immediately is to combine precision and recall using F1-measure and use this as a decision metric to pick the best model.

However the requirement I am given is that accuracy should also be part of the decision metric or I should prove that combining F1-measure and accuracy will not improve the decision metric. Does anybody have any idea how to do either?

Improve this question
$\endgroup$
3
  • $\begingroup$ is the dataset balanced? $\endgroup$
    – Aksakal
    Commented Sep 21, 2022 at 14:11
  • $\begingroup$ It may or may not be balanced. We can't know in advance. $\endgroup$
    – Emin Ozkan
    Commented Sep 22, 2022 at 0:42
  • $\begingroup$ Did you try auroc? $\endgroup$
    – Aksakal
    Commented Sep 22, 2022 at 13:17

2 Answers 2

Reset to default
5
$\begingroup$

This approach is questionable. Instead use establish statistical principles as detailed here. Here are a few.

  1. Turn the problem into a prediction problem instead of a classification problem, by estimating outcome probabilities. This allows for close calls, gray zones, etc., and does not require artificial "data amputation" by balancing. Imbalance is not a problem.
  2. Use the gold standard log likelihood or penalized log likelihood (or Bayesian posterior) measure which captures much more information than the discontinuous measures you mentioned.
  3. In addition to using the log likelihood as the optimality criterion, unbiasedly and flexibly estimate the calibration curve to show that the predicted risks have good absolute accuracy. This allows the predictions to be used for decision making.
Improve this answer
$\endgroup$
0
$\begingroup$

Although my manager's original request was to combine accuracy, precision and recall, I was able to sell him the idea that what we really want to do is to combine all four quadrants of the confusion matrix in a balanced way. What I mean by the balanced is that the metric should work under extreme class imbalances. After some research, I came up with two candidate metrics: Balanced accuracy and Matthews' Correlation Coefficient (MCC). They both work when there is class imbalance. Unlike these metrics accuracy fails whenever there is class imbalance; precision and recall fails when there are more positives than negatives(assuming that they are equally important to you).

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ What about proper scoring rules that evaluate the predicted probability values? $\endgroup$
    – Dave
    Commented Oct 22, 2022 at 14:57
  • $\begingroup$ @Dave Could you elaborate on what you mean? $\endgroup$
    – Emin Ozkan
    Commented Oct 23, 2022 at 15:23
  • $\begingroup$ fharrell.com/post/class-damage fharrell.com/post/classification $\endgroup$
    – Dave
    Commented Oct 23, 2022 at 16:00
  • 1
    $\begingroup$ From the business perspective, it's dangerous to consider the four quadrants of the confusion matrix this way unless you also consider the costs/benefits of making true and false class assignments of each type. See this page and its linked pages, for example. $\endgroup$
    – EdM
    Commented Oct 27, 2022 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.