I have 2 copora:

  • List 1 contains 1 million words in total. This represents general written usage.
  • List 2 contains 5 million words in total. This represents a sub-genre of literature.

I would like to isolate those words that are special to that genre; by comparing the two lists. Using some software I wrote, I can get data like this:

  • The word "elephant" appears 20 times in list 1.
  • The word "elephant" appears 100 times in list 2.
  • The word "forest" appears 20 times in list 1.
  • The word "forest" appears 300 times in list 2.

In this case, it seems clear to me that "elephant" is not so special to the genre (20/1000000 = 100/5000000), however, "forest" is probably special (20/1000000 < 300/5000000).

However, there are some cases where it seems more difficult to make a decision:

  • The word "bridge" appears 20 times in list 1.
  • The word "bridge" appears 150 times in list 2.
  • The word "raccoon" appears 1 time in list 1.
  • The word "raccoon" appears 10 times in list 2.

In the first example, "bridge" appeared more frequently in list 2, but is that difference significant? In the second example, "raccoon" appears 10 times more frequently in list 2, but given that the total occurrences are so low, is that significant?

  • How can I obtain a cut-off point for deciding which words are specific to the genre?
  • $\begingroup$ Are these lists of words, of phrases, sentences, paragraphs, entire documents, or something else? I think the answer is crucial. Consider, for example, an extreme case where corpus 1 consists of ten documents and corpus 2 of 50 documents. Perhaps one of the ten in corpus 1 mentions "forest," occurring 20 times therein, and five of the 50 in corpus 2 mention "forest," occurring from 20 to 150 times per document for a total of 300 occurrences. Now it appears there is no significant difference in usage of "forest" between the corpora. $\endgroup$ – whuber Nov 30 '11 at 17:28
  • $\begingroup$ I see, perhaps I will need to change my approach. The first corpus is similar to the Lancaster-Oslo-Bergen Corpus. It has several hundred 1,000 to 2,000-word texts scattered amongst various genres. Although the Web site hosting the corpus posted a frequency list, perhaps I cannot understand this as representative of general usage. This problem seems far more difficult than I have previously thought. $\endgroup$ – Village Dec 1 '11 at 12:34

In order to test for statistical significance, you need to have a model on which to base your test statistics. The Poisson, binomial and negative binomial distributions are commonly used to represent the distributions of count data when these are treated as random variables. You could then look at a statistical test suitable for the model. For example, Pearson's Chi-squared goodness of fit is probably suitable for your needs. Then you base your threshold on a certain significance level (e.g. p<0.01) or a range of significance levels (so you can observe the number of instances that hit significance at different levels.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.