0
$\begingroup$

I have a few thousand tagged documents that I use to train a Naive Bayes text classifier (using sklearn in this case). In addition to the tagged documents, I have about 100k untagged documents.

The accuracy is OK (~78%), but the model is overfitting the training data. I want to try to weigh the tfidf features after their frequency in the whole corpus before I feed the feature vector to the model. I'm sure there is a name for this procedure but I don't know what it is.

For instance, a specific word may be very rare in the training data but common in the untagged dataset. I want to take statistics such as that into account. How do I do that?

$\endgroup$
1
$\begingroup$

You should take a look at Transformed Weight Complement Naive Bayes.

https://people.csail.mit.edu/jrennie/papers/icml03-nb.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.