2
$\begingroup$

I have 3 different texts, and for each one of them, I have the frequencies of 25 letters appearing in those texts (Text A, letter A, freq = 5013; Text A, letter B, freq = 1462,...). These texts have different, and not normal distributions.

I would like to know, if there's any significant difference between the texts, in terms of how the letters are ranked by frequency (for example: letter A is the most frequent letter in texts A and B, but only 5th-most frequent in text C; letter E is 2nd-most frequent in text A, 3rd-most frequent in text B - almost no difference - and only 9th most frequent in text C; etc.).

I performed a Kruskal-Wallis test by ranks, and you can see the results of the test on some random texts below. What concerns me here are two things:

  • Kruskal-Wallis test calculates the p value depending on the mean ranks, so the actual order of items (in this case letters) is not important,
  • One of the assumptions of the test is that the distributions have to have the same shape (variability), which is not the case here.

Either way, the results on the sample data showed no significant difference, even though text A and B were texts from a book, and text C was a computer-generated text, and the ranks clearly differed (like for example, for letter A, ranks (and frequencies) are almost the same for text A and B, but completely different for text C).

I am wondering, if I'm even using the right test to test my hypothesis? In case I'm not - which test should I use?

enter image description here

$\endgroup$
0
$\begingroup$

I'm going to redo my answer. The old one, recommending a K-S test, was misguided (K-S tests are for continuous distributions, not categorical ones).

I'd recommend you use a chi^2 test. A transcript using R is shown below. Rank tests (Kruskal-Wallis and Mann-Whitney) appear to be rather weak on your data, and in any case are not very robust when the counts are so close that the ranks are untrustworthy (especially true of the 'C' counts).

A=c(5013,1462,492,1622,4730,132,556,343,5870,1430,1869,2022,1100,3517,4091,1711,2632,2245,3865,697,2053,1194,686,382,286)
B=c(5388,876,453,1690,4909,143,807,532,4627,2012,1982,2257,1456,3528,4411,1670,2922,2399,2350,966,2154,1033,672,520,320)
C=c(2444,2331,2239,2397,2245,2254,2321,2212,2275,2350,2409,2371,2403,2346,2357,2337,2313,2399,2372,2360,2305,960,0,0,0)

chisq.test( cbind(A,B) )
#
#         Pearson's Chi-squared test
#
# data:  cbind(A, B)
# X-squared = 1048.411, df = 24, p-value < 2.2e-16
#

chisq.test( cbind(A,C) )
#
#         Pearson's Chi-squared test
#
# data:  cbind(A, C)
# X-squared = 13443.61, df = 24, p-value < 2.2e-16
#

chisq.test( cbind(B,C) )
#
#         Pearson's Chi-squared test
#
# data:  cbind(B, C)
# X-squared = 11938.08, df = 24, p-value < 2.2e-16
$\endgroup$
  • $\begingroup$ Isn't a chi^2 test a bit too sensitive for this data, since it compares the differences for each letter and then adds them up? The test in your case shows sig. dif. for all three cases. Shouldn't it be sig. for A vs. C and B vs. C and not sig. for A vs. B, depending on the data? $\endgroup$ – NoelAramis Oct 8 '15 at 7:27
  • $\begingroup$ Plot the counts against each other: they are significantly different distributions. $\endgroup$ – Creosote Oct 8 '15 at 19:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.