1
$\begingroup$

I'm working on a multi-class multi-label classification problem where text (let's say comments on a website) should be assigned (possibly multiple) labels. There is a neutral (negative) class and there are 7 positive classes (e.g., aggressive, racist, sexually inappropriate, etc.). Also, as you would expect, the distribution of the data is extremely skewed towards the neutral class, i.e., there are very few inappropriate comments.

I'm trying to decide on a metric to be optimized. So, far I have decided that micro-averaged label-based metrics and example-based ones (e.g., Hamming loss) are not suitable as they give equal weight to all classes and tend to get overwhelmed by the model's performance on the majority class, whereas I care more about performance on minority classes (a good review here).

I believe that macro-F1 should be a sensible metric to optimize. However, a colleague has proposed macro-recall. I'm wondering if it would make any sense to try to optimize macro-recall in a multilabel setting.

So, here is my question:

Am I right in thinking that if Recall is to be optimized, one can hack the metric and achieve a perfect macro-averaged recall by predicting all the labels for all the test cases (assuming labels are not mutually exclusive)?

Put differently, wouldn't optimizing Recall induce the model to be overly aggressive in prediction and predict as many labels as it can for test cases.

$\endgroup$
1
$\begingroup$

Recall is usually defined by assigning each sample to a single class, then checking whether this assignment was correct or not. So I'm a bit unsure of what you mean by "predicting all the labels for all the test cases", and how you would define True/False Positive/Negative and Recall in this case.

Think of whatever decision your model will support. What decision would you take if your model said "this word is racist, or sexist, or whatever"?

In any case, I would use neither F$\beta$ nor recall. Both are improper scoring rules and suffer from exactly the same problems as covered in Why is accuracy not the best measure for assessing classification models? As I argue there and Frank Harrell does elsewhere, you should assess probabilistic predictions using proper scoring rules. The decision you make based on your model is not part of the modeling step any more, and needs to incorporate costs.

$\endgroup$
  • $\begingroup$ Many thanks for your answer. I believe in a multi-label setting, it is meaningful to assign all the labels to an example, is it not? (e.g. a comment is racist, sexist and aggressive, assuming 3 classes). And I'm asking if optimizing recall (without penalizing for low precision) would induce the model to do so. Just for reference, I am thinking of a multi-label recall as defined here on page 5: bit.ly/2V0RlBW. (true/false pos/neg are also defined on the same page). I'm reading Frank's posts now and will probably have more questions soon. Thanks. $\endgroup$ – Milad Shahidi Apr 15 at 14:51
  • 1
    $\begingroup$ Well, if you just want high recall, then a smart model would indeed assign all labels to all samples. It looks like you have a multivariate question. $\endgroup$ – Stephan Kolassa Apr 15 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.