In the task of language identification, I would do the following:
- Take a sample of my data
- Prepare the ground truth
- Train my classifier on this sample data
- Test classifier accuracy on the other part of my data that I did not sample (and possibly cross-validate)
Let us say I am working on identifying the language of a tweet. Getting the ground truth of this is hard (either I need to manually annotate the tweets with their correct language or crowd-source this task). Therefore, what I did was to train my classifier on already available text (such as eBooks, news articles) of different languages. This is plain wrong - I am training my classifier for one domain (large well-written text) but testing it on an other domain (small potentially-garbled up text). Surprisingly, this worked with a 90% accuracy on 400 hand-tagged tweets.
How will I know what is the sample size I need to pick (which is 400 in my case) that can show that the classifier will still behave at about 90% for a million tweets?