Optimising recall for multi-label classification?

Question

I'm working on a multi-class multi-label classification problem where text (let's say comments on a website) should be assigned (possibly multiple) labels. There is a neutral (negative) class and there are 7 positive classes (e.g., aggressive, racist, sexually inappropriate, etc.). Also, as you would expect, the distribution of the data is extremely skewed towards the neutral class, i.e., there are very few inappropriate comments.

I'm trying to decide on a metric to be optimized. So, far I have decided that micro-averaged label-based metrics and example-based ones (e.g., Hamming loss) are not suitable as they give equal weight to all classes and tend to get overwhelmed by the model's performance on the majority class, whereas I care more about performance on minority classes (a good review here).

I believe that macro-F1 should be a sensible metric to optimize. However, a colleague has proposed macro-recall. I'm wondering if it would make any sense to try to optimize macro-recall in a multilabel setting.

So, here is my question:

Am I right in thinking that if Recall is to be optimized, one can hack the metric and achieve a perfect macro-averaged recall by predicting all the labels for all the test cases (assuming labels are not mutually exclusive)?

Put differently, wouldn't optimizing Recall induce the model to be overly aggressive in prediction and predict as many labels as it can for test cases.

Answer 1

Recall is usually defined by assigning each sample to a single class, then checking whether this assignment was correct or not. So I'm a bit unsure of what you mean by "predicting all the labels for all the test cases", and how you would define True/False Positive/Negative and Recall in this case.

Think of whatever decision your model will support. What decision would you take if your model said "this word is racist, or sexist, or whatever"?

In any case, I would use neither F$\beta$ nor recall. Both are improper scoring rules and suffer from exactly the same problems as covered in Why is accuracy not the best measure for assessing classification models? As I argue there and Frank Harrell does elsewhere, you should assess probabilistic predictions using proper scoring rules. The decision you make based on your model is not part of the modeling step any more, and needs to incorporate costs.