# Multiple comparisons problem in corpus linguistics

A collocation is an expression consisting of two (or more) words that correspond to some conventional way of saying things, for example strong - tea or new - car.

We really want to know whether two words occur together more often than chance. Assessing this is the classical problem of statistic, for example we can use chi-square test for independence or Fisher exact test. Most papers on corpus linguistics formulate $H_0$ that there is no association between the words beyond chance occurrence, compute the probability and then reject $H_0$ if $p < 0.05$.

A 2-by-2 table below shows the dependence of occurrence of words new and car. There are 8 occurrences of new car in the corpus and 4600 occurrences where the second word in car, but the first word is not new, etc.

+------------+----------+----------+
|            | w1 = new | w1 ≠ new |
+------------+----------+----------+
| w2 = car   |        8 |     4600 |
+------------+----------+----------+
| w2 ≠ car   |    15300 |  1234567 |
+------------+----------+----------+

We then apply standard chi-square test for independence on such contingency table. We apply statistical test for all co-occurrence pairs in the corpus (e.g., for all nouns) and evaluate $p$-value for each pair. For example, the final table in corpus linguistics is often of the form

+--------+--------+-------+
| word 1 | word 2 | chisq |
+--------+--------+-------+
| word A | word B |  3.82 |
| word B | word C | 15.46 |
| word C | word D |  0.61 |
| etc.   | etc.   |  etc. |
+--------+--------+-------+

My question: should we apply correction for multiple testing in such scenario (and if yes, which procedure do you recommend)?