I'm trying to implement a naive Bayes classifier to classify documents that are essentially sets (as opposed to bags) of features, i.e. each feature contains a set of unique features, each of which can appear at most once in the document. For example, you can think of the features as unique keywords for documents.

I've closely followed the Rennie, et. al. paper at http://www.aaai.org/Papers/ICML/2003/ICML03-081.pdf, but I am running into a problem that doesn't seem to be addressed. Namely, classifying short documents are resulting in much higher posterior probabilities due to the documents having a smaller number of features; vice versa for long documents.

This is because the posterior probabilities are defined as (ignoring the denominator):

$$ P(class|document) = P(class) * P(document|class) $$

which expands to

$$ P(class|document) = P(class) * P(feature_1|class) * ... * P(feature_k|class) $$

From that, it's clear that short documents with fewer features will have higher posterior probabilities simply because there are fewer terms to multiply together.

For example, suppose the features "foo", "bar", and "baz" all show up in positive training observations. Then, a document with single feature "foo" will have a higher posterior probability of being classified in the positive class than a document with features {"foo", "bar", "baz"}. This seems counter-intuitive, but I'm not quite sure how to solve this.

Is there some sort of length normalization that can be done? One idea is to add the size of the document as a feature, but that doesn't seem quite right since results would then be skewed by the size of documents in the training data.


1 Answer 1


I don't think this should actually matter since classification is done by choosing the class with the maximum posterior compared to the other classes, not to other observations. Of course, I'm assuming each class uses the same features.

  • $\begingroup$ I'm actually looking to assign a score to each classification, so that I can use the score to order the documents. For example, it would be useful in a recommendations setting where I want to classify documents into those that the user will like and those that the user will dislike, then use the classification score to order the recommendations. $\endgroup$
    – pmc255
    Sep 10, 2011 at 3:34
  • $\begingroup$ The denominator you've ignored, $P(D)$, is the normalizing constant. Normally this is fine since it is constant across classes and thus doesn't effect the decision rule when classifying a single observation. Adding this term back in should help. $\endgroup$
    – alto
    Sep 10, 2011 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.