8
$\begingroup$

As a part of a project for the university is should train a Naive Bayes classifier to classify question and answers in three different categories, the task should be easy since that the 3 classes are really different between each other.

Dataset

the dataset is a mixture of questions and answers from different domains (furniture and a C++ course) and they are in different language (Italian and English) so at the first sight it should be easy to classify them.

The only problem with the dataset is that is really unbalanced, like C++ course 2700 training instance. furniture English 200 training instance. furniture Italian 60 training instance.

Feature Extraction

The feature are simply n-gram counts i don't remove stopword because i have 2 different language to work with.

Using TF-IDF features and stemmed token i obtained lower results.

The NB algorithm

I have implemented the naive bayes by myself but it obtains the same result of the scikit learn one. i have trained it with per class prior and a smoothing using alpha=.5

The results

The result at the end was in some sense good

              precision    recall  f1-score   support

      0           0.92      1.00      0.96       318
      1           1.00      0.50      0.67        44
      2           1.00      0.33      0.50         6

avg / total       0.93      0.93      0.92       368

But the only draw back is that the recall on the 1 and 2 class is low, and the reason is simple (we have unbalanced classes).

There is a way to actually obtain lower but more balanced results having the constraint of using a single model trained with Naive Bayes?

Cheers

Edit: My needs of having more balanced results is due to the fact that I should use this classifier as part of an Question Answering task.

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Just an idea: What about adding a classifier to predict whether a given question is english or italian ? This should be easy, isn't it ? Afterwards one does learn a separate classifier for the domain, one for each language. This way you can filter stopwords etc. C++ and furniture are sooo different, that I'd expect a good performance based on 1-grams (without stopwords ofc). Maybe a closer look which instances have been classified incorrectly might help ? $\endgroup$
    – steffen
    Commented May 22, 2014 at 13:36
  • $\begingroup$ Yes this was my first thought and, well a linear classifier from my point of view can perform even better, but the professor gave us this constraint to use an unique NaiveBayes classifier. $\endgroup$
    – dbonadiman
    Commented May 22, 2014 at 13:44

2 Answers 2

Reset to default
5
$\begingroup$

Tackling the Poor Assumptions of Naive Bayes Text Classiffiers suggests some modifications to Naive Bayes in order to correct for biased sample sets.

Also have a look at this (and similar) CV posts on class imbalance, unbalanced class labels, etc.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is really useful and i adapted my Naive-Bayes implementation to do the TWCNB as said in [1] but it was not successful, i think that the problem at the end is that the data are too much unbalanced so i may not achieve anything better using a single model trained with NaiveBayes. $\endgroup$
    – dbonadiman
    Commented May 22, 2014 at 11:25
  • $\begingroup$ @dbonadiman There is a python implementation of the adaption mentioned in the paper: sklearn.naive_bayes.ComplementNB $\endgroup$
    – Jan
    Commented Jul 12, 2023 at 15:14
0
$\begingroup$

I know an answer has already been accepted. i thought the following could help future reader.Try implementing this to handle unbalanced classes, This worked pretty good for me.

Naive Bayes for Text Classification with Unbalanced Classes

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ While your answer includes a nice citation, it is usually best if you include the most important parts from your source in your answer (so that everything is self-contained and does not rely on external inks remaining stable). $\endgroup$
    – Tavrock
    Commented Feb 27, 2017 at 6:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.