I'm trying to find words that have a significantly distribution in one small portion of a large corpus than they do in the entire corpus. I'm using the standard likelihood-ratio test statistic

$$\chi^2 = 2 \sum_{i} O_i log(O_i/E_i)$$

where the two classes are the small portion of the corpus and the rest of the corpus.

The problem that I run into is that the small portion of the corpus is so much smaller than the whole corpus that the expected number of occurrences of any word in that small portion is on the order of 0.001. Therefore, the fact that a word occurs at all in the small portion is taken to be very significant, even if it's a very common word like a, the, of, or another stopword.

Is there a way to do a better estimate that doesn't underestimate expected values so badly? Or would a different test be more appropriate here?

Thanks so much for your help.

  • 1
    $\begingroup$ Do you choose the words you suspect of being over-represented based on looking at the data, or do you have a hypothesis developed beforehand you want to check? Which way would make quite a difference to the approach. $\endgroup$ – Peter Ellis Jun 11 '12 at 6:20
  • $\begingroup$ Just how small is the small portion, i.e. how many words? $\endgroup$ – onestop Jun 11 '12 at 12:55
  • $\begingroup$ I want to choose the over-represented words based on the scores provided by the test, so that I can set some threshold and list of words that characterize the smaller segment. (The smaller segment by the way is usually on the order of 100 words, while the whole corpus is ~1,000,000) $\endgroup$ – Ben Jun 11 '12 at 13:57
  • $\begingroup$ I suspect that 100 words is just too small to identify over-represented words. $\endgroup$ – onestop Jun 11 '12 at 16:55

The likelihood ratio test is intended to test to see which model in hierarchical order better fits the data. it is not intended to be used the way you are using it. I think you want to compare two distributions which can be accomplished with a chi-square test, two-sample K-S test or other tests of that type.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.