0
$\begingroup$

For a classification problem (based on news summery, put news articles into 4 classes), we built a linear SVC model, trained with 120,000 data and tested with 7,600 data. The result is:

Accuracy = 0.9134210526315789 Weighted Precision = 0.9134148091865167 Weighted Recall = 0.9134210526315789 F1 = 0.9132905479246967

The question is how to evaluate such result? Is the result good, normal or bad?

Is there any rule of thumb value or anything we can compare, like famous classification problem result or kaggle competition result?

Thanks

$\endgroup$
  • $\begingroup$ Those numbers are rather high, but as you mentioned, it's all relative. Try and look for different work on a different dataset (perhaps in google scholar), and run your classifier on that dataset. Compare the results to the previous results. $\endgroup$ – Eran Sep 27 '17 at 7:45
1
$\begingroup$

If you don't have much experience with either the data or this type of task, I think @Eran's suggestion is correct.

Here are a few classic natural language processing text classification benchmarks:

Let's look at some old results:

  • In (Lewis et al., 2004) was getting a micro-averaged F1.0 score of 0.816 with SVMs on RCV1 on 101 topics.

  • In the scikit-learn documentation they get a macro-averaged F1 score of 0.769 with Naive Bayes based on 20 topics.

  • In (Li and Yang, 2004) get they get a macro-average F1 0.8857 with a SVM across 90 categories.

I don't know what the state of the art is for these datasets, but I am sure you could track it down. You can see also that there's quite a bit variation in terms of how hard these datasets are.

Depending on which dataset yours is more like, I'd interpret your results somewhere in the range of middle of the road to quite good. Now you could run your model on these datasets and directly compare how effective your models is on these known datasets. This is the common task framework methodology in action!

Lewis, David D., et al. "Rcv1: A new benchmark collection for text categorization research." Journal of machine learning research 5.Apr (2004): 361-397.

Li, Fan, and Yiming Yang. "A loss function analysis for classification methods in text categorization." Proceedings of the 20th International Conference on Machine Learning (ICML-03). 2003.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.