I'm working on a document binary classification problem where I have a decent sized corpus of about 30,000 documents (600-1000 words each). My approach is to select a sample of documents and manually classify them, create a naive Bayes model, and apply it to the rest of the data.

The biggest question is how to determine the number of documents to manually classify. Are there any rules of thumb?

  • $\begingroup$ It depends, how many categories do you have ? $\endgroup$ – RUser4512 Dec 28 '15 at 10:13
  • $\begingroup$ It's a binary problem so there are two. $\endgroup$ – Ellis Valentiner Dec 28 '15 at 13:37

There is no rule of thumb that I am aware of. But counting can provide some insight.

Calling :

  • $p$ the number of different words,
  • $l$ the number of words per document,

You would like your estimate of $P(X_i=1|word_j)$ to be evaluated many times (say 10). Indeed, this is what naive Bayes rely on to perform classification.

Therefore, you need about $10p$ observations. Each line brings you $l$ observations. Supposing you chose them as different as possible, you would like to have around $10p/l$ lines. In your case, this amounts to $p/100$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.