# Statistical significance between letter frequencies in different corpora

I have letter frequencies from two corpora. What methods I can use to test:

1. if there is statistical significance between the frequencies in general
2. if there is statistical significance between the frequencies of a particular letter (for example, A)

Sample data:

    Corpus_1    Corpus_2
a   2367073     467680
b   364195      73234
c   244794      42309
d   617722      136303
e   1316156     262776
f   52370       8329
…   …           …
z   393095      70522


The frequencies in general can be tested via a chi-square test of indepence. In R:

letters <- read.table(header=TRUE, text="Letter Corpus_1    Corpus_2
a   2367073     467680
b   364195      73234
c   244794      42309
d   617722      136303")

chisq.test(letters[,c("Corpus_1", "Corpus_2")])


However, due to the large numbers, this will become significant for the slightest deviation:

> chisq.test(letters[,c("Corpus_1", "Corpus_2")])

Pearson's Chi-squared test

data:  letters[, c("Corpus_1", "Corpus_2")]
X-squared = 1907.3, df = 3, p-value < 2.2e-16


For single letters there is the problem, that one corpus may contain more letters overall than the other. You could use a test of proportions (e. g. chisquare test again) of the proportion of letter a in all letters in corpus A against the same proportion in the other corpus.

Let's say, in corpus A are 1 million letters that are not b and in corpus B there are 2 million letters that are not b, than:

> b <- matrix(c(364195, 1000000, 73234, 2000000), nrow=2)
> b
[,1]    [,2]
[1,]  364195   73234
[2,] 1000000 2000000
> prop.test(b)

2-sample test for equality of proportions with continuity
correction

data:  b
X-squared = 397530, df = 1, p-value < 2.2e-16
alternative hypothesis: two.sided
95 percent confidence interval:
0.4980179 0.5004771
sample estimates:
prop 1    prop 2
0.8325808 0.3333333


If you test for each letter, than that is a lot of tests. Consider Bonferroni correction or something similar.

For both of your questions, you could use a Chi-square test (or Fisher's exact test) to look for statistical differences.

if there is statistical significance between the frequencies in general

To address this you would simply compare all of the counts from corpus 1 with all of the counts in corpus 2. If the test is significant then that tells you that the distribution of letters (i.e. the relative counts of each letter) varies between the two corpora.

if there is statistical significance between the frequencies of a particular letter (for example, A)

One approach here would be to calculate two numbers for each corpus. Firstly, the number of As. Secondly, the number of all other letters (i.e. B-Z). If you again use the Chi-square test then a significant result will tell you that the frequency of A varies between the two corpora. If you plan to do this for many letters then you may want to consider some sort of correction to compensate for the multiple comparisons.

Another, more exploratory, approach would be to examine the standardised residuals from the Chi-square test you use to address the first question. For each letter you could look to see whether the standardised residual is greater than 2, which might indicate a significant difference. This approach is discussed in this article:

@article{test2015your,
title={Your Chi-Square Test is Statistically Significant: Now What?},
author={Test, Omnibus},
journal={Practical Assessment, Research \& Evaluation},
volume={20},
number={8},
pages={2},
year={2015}
}