Weighting common performance metrics by classification outcomes?

Question

Cost-sensitive classification metrics are somewhat common (whereby correctly predicted items are weighted to 0 and misclassified outcomes are weighted according to their specific cost). Some examples of implementations: mlr, tidymodels, costcla. For example:

$$ outcomeWeights = \left(\begin{array}{cc} 0 & 1\\3 & 0 \end{array}\right)$$

However other types of 'value' based weighting of the cells of confusion matrices seem to be less common. I have two questions:

1. Are other types of cell weights on a confusion matrix appropriate? (E.g. weighting based on the value associated with the cell not strictly the cost)
2. If yes, are weighted versions of common classification metrics appropriate to calculate (e.g. precision, recall, auc, etc.)?

For example: $$ outcomeWeights = \left(\begin{array}{cc} 0.14 & 1.89\\1.86 & 0.11 \end{array}\right)$$

Example

Say for instance we first generate a confusion matrix on a set of predictions

``` r
library(tidyverse)

set.seed(1234)

df <- tibble(class = c(rep(TRUE, 100), rep(FALSE, 100)),
             pred_prob = runif(200, 0, 1)) %>% 
  mutate(pred_class = pred_prob > 0.5) %>% 
  select(class, contains("pred")) %>% 
  mutate(across(contains("class"), factor, c("TRUE", "FALSE")))

confusion_matrix <- yardstick::conf_mat(df, class, pred_class)

confusion_matrix
#>           Truth
#> Prediction TRUE FALSE
#>      TRUE    45    52
#>      FALSE   55    48
```

But for business reasons we apply the weights mentioned previously

```r
outcome_weights <- matrix(
  c(0.14, 1.89, 1.86, 0.11),
  ncol = 2,
  byrow = TRUE
  )

confusion_matrix$table <- confusion_matrix$table * outcome_weights

confusion_matrix
#>           Truth
#> Prediction   TRUE  FALSE
#>      TRUE    6.30  98.28
#>      FALSE 102.30   5.28
```

Would common classification metrics be appropriate to review still (now in a weighted form)? For example at a specific decision threshold:

```r
summary(confusion_matrix)
#> # A tibble: 13 x 3
#>    .metric              .estimator .estimate
#>    <chr>                <chr>          <dbl>
#>  1 accuracy             binary        0.0546
#>  2 kap                  binary       -0.890 
#>  3 sens                 binary        0.0580
#>  4 spec                 binary        0.0510
#>  5 ppv                  binary        0.0602
#>  6 npv                  binary        0.0491
#>  7 mcc                  binary       -0.891 
#>  8 j_index              binary       -0.891 
#>  9 bal_accuracy         binary        0.0545
#> 10 detection_prevalence binary        0.493 
#> 11 precision            binary        0.0602
#> 12 recall               binary        0.0580
#> 13 f_meas               binary        0.0591
```

Or across thresholds as in weighted AUC or ROC curve or precision-recall curves?

Does the fact that the sum of the cells of the confusion matrix is different from the observation count make these weighted metrics inappropriate (and also that the sum of cell counts can change depending on the decision threshold)?

For example in the unweighted example we have a sum across cells of 200 (the number of observations) whereas in the second case we have 207. The number of 'weighted' observations hence will vary between decision thresholds (shown below):

```r
weight_cells <- function(confusion_matrix, weights = matrix(rep(1, 4), nrow = 2)){
  
  confusion_matrix_output <- confusion_matrix
  confusion_matrix_output$table <- confusion_matrix$table * weights
  confusion_matrix_output
}

conf_mat_threshold <- function(df_input = df, 
                               threshold = 0.5){
  hard_pred <- df_input %>%
    mutate(pred_class = pred_prob > threshold) %>% 
    mutate(across(contains("class"), factor, c("TRUE", "FALSE")))
  
  yardstick::conf_mat(hard_pred, class, pred_class)
}

thresholds <- tibble(threshold = c(0, unique(df$pred_prob), 1)) %>% 
  arrange(threshold)

conf_matrix_thresholds <- thresholds %>% 
  mutate(conf_mat = map(threshold, conf_mat_threshold, df_input = df)) %>% 
  mutate(conf_mat_weighted = map(conf_mat, weight_cells, weights = outcome_weights))

conf_matrix_thresholds %>% 
  mutate(obs = map_dbl(conf_mat_weighted, ~sum(.x$table))) %>%
  mutate(across(contains("conf"), list(obs = ~map_dbl(.x, ~sum(.x$table))))) %>% 
  ggplot(aes(x = threshold, y = conf_mat_weighted_obs))+
  geom_line()+
  labs(title = "'Observation' counts by threshold",
       subtitle = "After applying weights to confusion matrix")
```

Answer 1

Differential costs and benefits of class assignments are part of the reason why measures like accuracy are considered poor ways to assess models. If you have a well-calibrated model of class probabilities, the cost issues that you raise in this two-class scenario reduce to a simple choice of a cost-based probability cutoff for class assignment.

Let's say the cost of assigning a case to class $i$ when it's actually in class $j$ is $c_{i|j}$. Thus you can assign costs for correct classifications, and those "costs" could be negative (representing a gain).

Start with a simple situation: classes labeled $1$ and $2$, and no cost for correct classification. (That means values of 0 along the diagonal of your cost matrix.) Say that the estimated probability for assignment of a case with predictor values $x$ to class $1$ is $\hat p_1(x)$. Then with only $c_{1|2}$ and $c_{2|1}$ to consider as costs, to minimize cost you need to determine which of the expected costs, $c_{2|1}\hat p_1(x) $ or $ c_{1|2}(1-\hat p_1(x))$ is lower. Those costs are equal when:

$$ \hat p_1(x) = \frac{c_{1|2}}{c_{1|2}+c_{2|1}},$$

providing a cost-adjusted decision threshold.

For the two-class setting and non-zero (potentially negative) costs for correct assignments, the same argument provides a threshold of:

$$ \hat p_1(x) = \frac{(c_{1|2}-c_{2|2})}{(c_{1|2}-c_{2|2})+(c_{2|1}-c_{1|1})},$$

Notice that has the same form as for the simpler case without costs for correct classifications. So you can just re-define the misclassification cost as the difference in cost between incorrect and correct class assignment given the true class membership, set the cost of correct classification to 0, and get the same cost-based decision threshold. That's probably why you always see values of 0 along the diagonal of two-class cost matrices; from the decision-threshold perspective, you can always redefine the costs to achieve that.

Things are more complicated in multi-class situations, as explained by O'Brien, Gupta and Gray in Proceedings of the 25th International Conference on Machine Learning, Helsinki, Finland, 2008. The above is based on their presentation of the two-class case.

On this site you will see much emphasis on well-calibrated models and proper scoring rules instead of the measures that you report for your model. Rather than trying to figure out ways to adjust your measures to represent differential costs and benefits, it's simpler to get the probability model right first and take the costs/benefits into account at the ultimate class-assignment step. That said, there is a whole universe of potential proper scoring rules. If you already have a sense of the relative costs and benefits, you can choose a proper scoring rule that emphasizes probabilities near those of the cutoff you will ultimately use. Yi Shen describes this approach in his 2005 University of Pennsylvania thesis; this answer discusses that and related approaches like targeted maximum-likelihood.

Answer 2

A problem with weighting these metrics is that they are typically interpreted against "no-information" models. For example AUC is the probability an (actual) positive case gets a higher prediction than a negative case, which (assuming no information in the model) is generally 0.5. The "no information" state for the precision recall curve is defined by the level of class imbalance (resource).

Hence, when the cells of the confusion matrix are being weighted these baseline comparisons will fluctuate with the decision threshold. This makes interpretations unintuitive and therefore the weighted metrics not particularly useful.

As an example, let's plot the ROC curve from the original question. The predictions were random so we would expect them to align (roughly) with the theoretical line of no information (the blue dashed line). In the unweighted case, we roughly see this:

```{r}
# See code in Original Question for how conf_matrix_thresholds is generated
conf_matrix_summary <- conf_matrix_thresholds %>% 
  mutate(across(contains("conf"), list(metrics = ~map(.x, summary))))

conf_matrix_summary %>%
  select(threshold, conf_mat_metrics) %>% 
  unnest() %>% 
  filter(.metric %in% c("sens", "spec")) %>% 
  spread(.metric, .estimate) %>% 
  ggplot(aes(x = 1 - spec, y = sens))+
  geom_line()+
  geom_abline(slope = 1, colour = "blue", linetype = "dashed")+
  labs(title = "'Observation' counts by threshold",
       subtitle = "After applying weights to confusion matrix")
```

However after weighting, we see these no longer line-up.

```{r}
conf_matrix_summary %>%
  select(threshold, conf_mat_weighted_metrics) %>% 
  unnest() %>% 
  filter(.metric %in% c("sens", "spec")) %>% 
  spread(.metric, .estimate) %>% 
  ggplot(aes(x = 1 - spec, y = sens))+
  geom_line()+
  geom_abline(slope = 1, colour = "blue", linetype = "dashed")+
  labs(title = 'Comparing "no-information" states',
       subtitle = 'Confusion matrices have been weighted')
```

Making any interpretations difficult. (Similar critiques could be made against trying to weight other metrics commonly used to evaluate classification models.)