I'm trying to extract collocations from some text data, and I use statistical tests to tell if an n-gram is a good collocation candidate or not.
All the sources I came across so far (for example, Manning's book on Statistical NLP, chapter 5) only treat 2-gram candidates and do not go on to considering 3-grams and higher order compound terms.
For bigrams it works as following:
- Under $H_0$ we assume that words $w_1, w_2$ are "independent", i.e. do not form a collocation. That is, $P(w_1, w_2) = P(w_1) \, P(w_2)$.
- Then we estimate $P(w_1), P(w_2)$ and $P(w_1, w_2)$ from our data (e.g. by using Maximum Likelihood Estimator)
- Finally, we compare the observed value of $P(w_1, w_2)$ with the value expected under $H_0$.
How this can be extended to 3-grams and 4-grams? And how one would go about estimating probabilities for such n-grams under $H_0$?
Should $H_0$ for 3-grams look like $P(w_1, w_2, w_3) = P(w_1, w_2) \, P(w_3)$? Or rather $P(w_1, w_2, w_3) = P(w_1) \, P(w_2) \, P(w_3)$?