I am working on a classification problem, predicting if people travel overseas or not in the future, and have been using cross-validation to tune a model. I'm trying to decide on the best metric to use to assess each model, based on my particular classification problem. Here's an example confusion matrix:

                1        0
      1      9680     33021
      0     31003   3223439 

The binary response variable is unbalanced at a ratio of roughly 99:1. This shouldn't be a problem for the model as the training dataset is very large, but I think it may be relevant for the choice of metric.

The criteria I need the model to fulfil are:

  1. It predicts the correct number of the rare cases i.e Predicted condition positive (PP) is equal to the actual condition positive (P)
  2. The predictions of rare cases are as accurate as possible - i.e. that the proportion of the Predicted condition positive (PP) that are True Positives (TP) is as high as possible.

Many available metrics seem to cover No. 2 - for example, in this case I think either Sensitivity or Precision will tell me how accurate the rare case predictions are.

However, very few existing metrics seem to be concerned with No.1 - predicting the correct number of cases.

I wondered if anybody had experience of trying to optimise a model based on these criteria. Is there a single metric I could use that would give a combined score for these two measures? Or, is it likely that there's a trade-off between the two, so that a model that is good at (1) is less good at (2) and vice versa. Anecdotally, the results of my various model runs so far are consistent with such a trade-off - however I don't really understand the reason for one existing conceptually, so any explanation would be useful.

Many thanks

  • $\begingroup$ Why is #1 not just accuracy? For #1, do you really care about the counts, and not whether they’ve been assigned to the correct examples? $\endgroup$ Commented May 21, 2021 at 13:40
  • $\begingroup$ Yes, the count is the important thing - the forecasts are for financial planning purposes, and people want an idea of the overall count of these cases. $\endgroup$
    – rw2
    Commented May 21, 2021 at 13:46


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.