I study linguistics, and I'm conducting a research which compares the ratio between two categories of words in 2 different text corpuses.

Let's say I have corpuses A and B. I created 2 categories of words, Let's call them C and D. I want to calculate the ratio between the count of words in category C and the count of words in category D, and see if the ratio in corpus A is significantly larger than the ratio in corpus B.

What statistical test can I use to test this hypothesis?

It's worth mentioning that the count of words in category C should theoretically depend on the count of words in category D and vice versa.

Thanks in advance


First, note that if the ratio of C to D is larger in corpus A, then so will be the proportion. So you can do the standard hypothesis test for difference of proportions.

Let: $$p_A = {C_A\over C_A + D_A}$$ $$p_B = {C_B\over C_B + D_B}$$ $$p = {C\over C + D}$$ These are the proportions of words in each corpus as well as the total for both corpuses.

Then set $$Z = {(p_A - p_B)\over p(1 - p)({1\over C_A + D_A} + {1\over C_B + D_B})}$$

If the number of words from categories C and D is large, then your test statistic should be approximately normal. The null hypothesis is that Z = 0 and the two proportions are equal. If you want a p-value of .05, you can reject the null hypothesis when Z > 1.645.

  • $\begingroup$ I'm not sure if I can use this test, since i'm not sure if the proportions are normally distributed. $\endgroup$
    – lior_
    Aug 1 '16 at 13:38
  • 1
    $\begingroup$ By the central limit theorem, the sampling distribution of a proportion will converge to being normally distributed. Assuming these are large corpuses, the test results using the normality approximation will be almost identical to what the exact results would be. That's why this is the default approach to testing the difference between proportions (see onlinecourses.science.psu.edu/stat414/node/268). If you have at least 50 or so words in each category, the normality approximation will be nearly exact. $\endgroup$
    – Wart
    Aug 1 '16 at 14:17
  • $\begingroup$ I think that should do it. But theoretically speaking, what if I have only 2 words in each category? $\endgroup$
    – lior_
    Aug 1 '16 at 14:46
  • $\begingroup$ You can use Fisher's exact test. See this question: stats.stackexchange.com/questions/123609/… $\endgroup$
    – Wart
    Aug 1 '16 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.