How to evaluate classification results (Precision, Recall, F1)?

Question

For a classification problem (based on news summery, put news articles into 4 classes), we built a linear SVC model, trained with 120,000 data and tested with 7,600 data. The result is:

Accuracy = 0.9134210526315789 Weighted Precision = 0.9134148091865167 Weighted Recall = 0.9134210526315789 F1 = 0.9132905479246967

The question is how to evaluate such result? Is the result good, normal or bad?

Is there any rule of thumb value or anything we can compare, like famous classification problem result or kaggle competition result?

Thanks

Answer 1

If you don't have much experience with either the data or this type of task, I think @Eran's suggestion is correct.

Here are a few classic natural language processing text classification benchmarks:

• Reuters-21578
• RCV1
• the 20 newsgroups dataset

Let's look at some old results:

• In (Lewis et al., 2004) was getting a micro-averaged F1.0 score of 0.816 with SVMs on RCV1 on 101 topics.

• In the scikit-learn documentation they get a macro-averaged F1 score of 0.769 with Naive Bayes based on 20 topics.

• In (Li and Yang, 2004) get they get a macro-average F1 0.8857 with a SVM across 90 categories.

I don't know what the state of the art is for these datasets, but I am sure you could track it down. You can see also that there's quite a bit variation in terms of how hard these datasets are.

Depending on which dataset yours is more like, I'd interpret your results somewhere in the range of middle of the road to quite good. Now you could run your model on these datasets and directly compare how effective your models is on these known datasets. This is the common task framework methodology in action!

Lewis, David D., et al. "Rcv1: A new benchmark collection for text categorization research." Journal of machine learning research 5.Apr (2004): 361-397.

Li, Fan, and Yiming Yang. "A loss function analysis for classification methods in text categorization." Proceedings of the 20th International Conference on Machine Learning (ICML-03). 2003.