I systematically chose rare words from three groups of texts (#1, #2 and #3) (the rare words within each group are different, but some rare words may appear in more than one group).

Then, for each rare word, I attempted to found out how frequent it is in one corpus (a large, systematic aggregation of texts), called Corpus P, and how frequent it is in another (Corpus B). Using the frequencies, I found out the log-likelihood statistic for each ratio of frequencies. This is a common statistic in corpus linguistics, and the bigger it is, the bigger the measure of "surprise" in the frequency ratio.

I was particularly interested in the rare words that were more significantly more frequent in Corpus P than in Corpus B. I want to see if the scores of rare words which meet this requirement significantly differ by text group (#1-3).

The scores for this set of rare words can be found, by text group and in descending order, in this dataset:

  1. Group #1 (n = 1362)
  2. Group #2 (n = 285)
  3. Group #3 (n = 112)

I looked at Q-Q plots at each of the series and they certainly are not normally distributed. Then, based on a literature review, I reached the hypothesis that these statistics have a chi-square distribution, I believe with one degree of freedom.

I wish to examine this assumption, and then find out if these are three significantly different series, i.e. if the rare words from each group have significantly different scores. (I tried to see if they were different without this assumption by applying a non-parametric Wilcoxon rank sum test, but the result was not significant.)

How do I see if each column in the data has a distribution similar to chi square, 1 d.f.? If I see that they are each a good fit to the model, how do I find out if the series are different from eachother?

Thanks in advance.


Looking at your data, it clearly isn't distributed $\chi^2(1)$; the numbers are way too large, as, for example, the $99.8^\text{th}$ percentile of a $\chi^2(1)$ distribution is about 9.5, and roughly 90% of your observations are greater than 10, with maxima in the thousands. Aside from this, I am not sure why you would think a $\chi^2(1)$ would be appropriate; the $\chi^2$ is a continuous distribution, but frequency counts are discrete, and at the low end of their range can only take on values such as 0, 1, 2..., which are evidently not drawn from a continuous distribution with mean 1.

Even if you divide each column by its mean, which scales it to the correct mean for a $\chi^2(1)$, a Kolmogorov-Smirnov test will easily reject the hypothesis that a $\chi^2(1)$ is appropriate, with results like the following:

    One-sample Kolmogorov-Smirnov test

data:  Group2 
D = 0.1879, p-value = 3.603e-09
alternative hypothesis: two-sided 

Warning message:
In ks.test(Group2, "pchisq", 1) :
  ties should not be present for the Kolmogorov-Smirnov test

As to whether the distributions are the same - I cannot see why they would be, based upon the data generating mechanism. The frequencies obviously depend on the size of the corpus. Just plotting histograms of the data will convince you that they don't have the same distribution. I also note your data is sorted; as a result, we don't know which word has which frequency in each corpus. Given your data, it is not possible to test the hypothesis that the words have the same probability of occurring in each corpus, which is what I assume you really want to test.

However, even with the data at hand, if we want to be formal about it, and compare the empirical distributions of the scaled data, we can again use a Kolmogorov-Smirnov test, which will only provide approximate p-values because of ties in the data. We get (deleting the warning messages about ties to save space):

> ks.test(Group1,Group2)

    Two-sample Kolmogorov-Smirnov test

data:  Group1 and Group2 
D = 0.1002, p-value = 0.01756

> ks.test(Group1,Group3)

    Two-sample Kolmogorov-Smirnov test

data:  Group1 and Group3 
D = 0.221, p-value = 8.123e-05

> ks.test(Group2,Group3)

    Two-sample Kolmogorov-Smirnov test

data:  Group2 and Group3 
D = 0.1798, p-value = 0.01107

Normally we wouldn't want to run three tests without correcting for multiple testing effects (e.g., if you run 100 tests at the 95% level of confidence, (very) roughly 5% will reject the null hypothesis by chance), the three tests are obviously not independent, and they are biased slightly towards accepting the null hypothesis (as we have fixed all three means to equal 1), but in this case it seems clear that we can reject the hypothesis that all three distributions are the same without further work.

  • $\begingroup$ Thanks for your help! "I also note your data is sorted; as a result, we don't know which word has which frequency in each corpus." - This is on purpose, I have this data saved separately. "Given your data, it is not possible to test the hypothesis that the words have the same probability of occurring in each corpus, which is what I assume you really want to test." - I have accomplished that with the actual Log Likelihood scores. I just want to know how if the scores are distributed differently. Why did we divide by the mean? Could the K-S test work with the raw data too? If not, why not? $\endgroup$ – Aviv May 26 '12 at 14:52
  • $\begingroup$ Because of the different corpus sizes, the frequencies are basically guaranteed to be different; you can use a K-S test on the raw frequencies but it will reject for sure, even if the underlying probabilities are the same. I was dividing by the mean just to get all the frequencies on the same scale - correct for different corpus sizes, if you will. $\endgroup$ – jbowman May 26 '12 at 15:02
  • $\begingroup$ OK. I already did that. (That info is required to get the Log Likelihood score for each word.) $\endgroup$ – Aviv May 26 '12 at 16:43

That is a very inappropriate way to frame a question for a statistics problem. Statistics is not about massaging data to find a way to make a result significant. The point of statistics is to collect scientific evidence to objectively test your hypothesis and lrt the evidence stermine whether you are right or wrong. So let me frame your problem in a way that is appropriate for statistical analysis. "I have three series that I think might represent different ways that they score word frequencies. The series show how word are scored in terms of their relative frequencies in a sample writing. They include many different words but also provide frequencies for the same words. What is an appropriate statistical method to try to show that the scoring methods are different?" This is a valid question but is a guess as to what you meant. Please make the question clearer and provide more detals about what these series are and what information in them helps demonstrate that they score differently.

  • $\begingroup$ Point well taken and question edited appropriately. $\endgroup$ – Aviv May 26 '12 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.