Document classification sample size

I'm working on a document binary classification problem where I have a decent sized corpus of about 30,000 documents (600-1000 words each). My approach is to select a sample of documents and manually classify them, create a naive Bayes model, and apply it to the rest of the data.

The biggest question is how to determine the number of documents to manually classify. Are there any rules of thumb?

• It depends, how many categories do you have ? Dec 28, 2015 at 10:13
• It's a binary problem so there are two. Dec 28, 2015 at 13:37

• $p$ the number of different words,
• $l$ the number of words per document,
You would like your estimate of $P(X_i=1|word_j)$ to be evaluated many times (say 10). Indeed, this is what naive Bayes rely on to perform classification.
Therefore, you need about $10p$ observations. Each line brings you $l$ observations. Supposing you chose them as different as possible, you would like to have around $10p/l$ lines. In your case, this amounts to $p/100$.