1
$\begingroup$

In the task of language identification, I would do the following:

  1. Take a sample of my data
  2. Prepare the ground truth
  3. Train my classifier on this sample data
  4. Test classifier accuracy on the other part of my data that I did not sample (and possibly cross-validate)

Let us say I am working on identifying the language of a tweet. Getting the ground truth of this is hard (either I need to manually annotate the tweets with their correct language or crowd-source this task). Therefore, what I did was to train my classifier on already available text (such as eBooks, news articles) of different languages. This is plain wrong - I am training my classifier for one domain (large well-written text) but testing it on an other domain (small potentially-garbled up text). Surprisingly, this worked with a 90% accuracy on 400 hand-tagged tweets.

How will I know what is the sample size I need to pick (which is 400 in my case) that can show that the classifier will still behave at about 90% for a million tweets?

Improve this question
$\endgroup$
5
  • $\begingroup$ I don't think what you are observing has anything to do with same size. I don't think there is a sample size answer to your question. In principle it is wrong to test a classifier on data with one set of class conditional distribution and expect it to work with any accuracy on a test set from a highly different set of class conditional distributions. If your result is correct there must be something special that makes the classes separate in a simlar way. I think there must be similarities in the two domains with respect to the tweet language. $\endgroup$
    – Michael R. Chernick
    Commented Sep 21, 2012 at 17:19
  • $\begingroup$ @MichaelChernick: Thank you. One reason for the good result could be the commonality of words in tweets and general language usage. Is there a way to say something like "it worked on a sample size of X, so with a confidence of 95%, it will work on the others"? $\endgroup$
    – Legend
    Commented Sep 21, 2012 at 19:12
  • $\begingroup$ I don't think so. The point I was making is that this probably has nothing to do with sample size. You just can't make statements like that. The issue is similarity of distributions. Can you get random samples from tweets and general usage words and compare their frequency distributions? There are formal statistical test that can address how similar two distributions are. You could then do inference and say whether or not the distributions can be seen to differ significantly. Showing that they don't differ would indicate why the approach you took worked in this case. $\endgroup$
    – Michael R. Chernick
    Commented Sep 21, 2012 at 19:34
  • $\begingroup$ I think you want to generalize your result. I don't really see how you can. What exactly did you mean when you wrote "It will work on the others" ? $\endgroup$
    – Michael R. Chernick
    Commented Sep 21, 2012 at 19:36
  • $\begingroup$ @MichaelChernick: Your suggestion makes sense - the one about frequency distributions. I'll try that out tonight. Regarding your other question, I meant to say that "because the classifier has X% accuracy on this sample set, it will perform sufficiently well with 95% confidence on other tweets that come from a similar distribution" $\endgroup$
    – Legend
    Commented Sep 21, 2012 at 21:13

0

Reset to default

Browse other questions tagged or ask your own question.