I'm working on a document binary classification problem where I have a decent sized corpus of about 30,000 documents (600-1000 words each). My approach is to select a sample of documents and manually classify them, create a naive Bayes model, and apply it to the rest of the data.

The biggest question is how to determine the number of documents to manually classify. Are there any rules of thumb?

  • $\begingroup$ It depends, how many categories do you have ? $\endgroup$
    – RUser4512
    Dec 28, 2015 at 10:13
  • $\begingroup$ It's a binary problem so there are two. $\endgroup$ Dec 28, 2015 at 13:37

1 Answer 1


There is no rule of thumb that I am aware of. But counting can provide some insight.

Calling :

  • $p$ the number of different words,
  • $l$ the number of words per document,

You would like your estimate of $P(X_i=1|word_j)$ to be evaluated many times (say 10). Indeed, this is what naive Bayes rely on to perform classification.

Therefore, you need about $10p$ observations. Each line brings you $l$ observations. Supposing you chose them as different as possible, you would like to have around $10p/l$ lines. In your case, this amounts to $p/100$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.