0
$\begingroup$

have three classes for sentiment (negative, neutral, and positive). I created synthetic fake data for the positive class the analogy now is 50% neutral, 45% positive, 5% negative. I get the metrics below and I am not 100% sure how to interpret them and if it is good to deploy this to production. I want the model to catch the positives and neutrals and not misclassification on negative class (aka very small False Positive on Negative class I suppose). How would you interpret this table?

0=negative, 1=neutral 2=positive but model trained on one hot encoding.

Classification Report

        precision    recall  f1-score   support

     0       0.06      0.93      0.10       643
     1       0.95      0.16      0.27     36755
     2       0.06      0.62      0.11      2309

accuracy 0.20 39707

macro avg 0.35 0.57 0.16 39707

weighted avg 0.88 0.20 0.26 39707

Compared to this

Classification Report

 precision    recall  f1-score   support  

0     0.40      0.45      0.43        44

1     0.90      0.87      0.88       751

2     0.46      0.52      0.49       123

accuracy 0.80 918

macro avg 0.58 0.61 0.60 918

weighted avg 0.81 0.80 0.81 918

$\endgroup$
10

1 Answer 1

1
$\begingroup$

" I want the model to catch the positives and neutrals and not misclassification on negative class (aka very small False Positive on Negative class I suppose). "

The results you have are probably near optimal results assuming that you have no preferences on the types of error that the model makes. If that assumption is incorrect (as it appears to be in this case), the results are not likely to be meaningful.

If you have particular concerns about one class, then that is an indication that the misclassification costs for your problem/analysis are not equal. What you should probably do is to work out what the misclassification costs are and build those into your classification, using "minimum risk classification" (also known as "cost sensitive learning", "Bayesian decision theory" etc).

A sort of agree with some of the comments saying not to use accuracy as a performance metric. It is based on the assumption that all misclassifications are equally bad, which is not always true, and a focus on accuracy tends to stop people from properly considering the misclassification costs, which can be of vital importance, especially for "imbalanced learning" tasks (as the minority class is often more "important" in some sense, and it is worth suffering some additional misclassifications of the majority class in order to catch more of the minority class).

I disagree with not using metric based on hard classifications though. If your application is one where you must make a hard classification, then it is likely that your metric of primary interest is based on that hard classification, and if you don't use it, you will be ignoring the primary goal of the project. You do need to be aware, however, of the shortcomings of these metrics, just as you need to be aware that proper scoring rules are no panacea either (see my answer here for an example where proper scoring rules chose the wrong model). That doesn't mean that it should be the only metric you use - sometimes it is good to use a variety of metrics to get an appreciation of different aspects of the model's performance.

So to summarise - work out plausible numeric costs for each type of misclassification and build that into your decision rule (rather than just picking the class with the highest probability).

$\endgroup$
2
  • $\begingroup$ But i do not use only F1 or only Accuracy, there is a mattrix and other values.I want to increase precision on Negative otherwise increase recall in Neutral and Positive. $\endgroup$ Aug 29, 2022 at 22:04
  • $\begingroup$ @nikolaosmparoutis I think my point boils down to the fact that you need to know what it is you are trying to do and know what your performance criteria are measuring and why they are appropriate for your problem. Having lots of performance metrics is not always a good thing. $\endgroup$ Aug 29, 2022 at 22:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.