As some of you may now, various statistical tests are used in the task of finding collocations in given text(either a single document or corpus of texts). Chi-square, t-test, likelihood ratio(with binomial distribution assumed). These tests are used to check if two words occur together significantly more often that apart from each other, that's how the strenth of collocation is measured.

A-a-anyway, one good thing to do is put your hands onto a large corpus of texts and save contingency tables(or just frequencies, really) of all the bigrams(two words) and unigrams(one word, obviously) from this corpus. We call this information background.

Little introduction for you there, now to the main part: when you receive a text(call it foreground), you extract all bigrams from it and have to decide whether this is a collocation or just some pair of words, like "of the" or "and yes", which do not bear any meaning and not at all important in this particular text.

Now, to the question: when you stumble upon a bigram that has in fact been seen in background, it means that both unigrams it consists of can be found in background too, all frequencies are in place, everything else is a piece of cake.

But what if this bigram has never occurred before? The only place we can find it in is our foreground, but there's not enough information to infer the distribution and collocation strength. I mean, you can apply any of these statistical tests to foreground contingency table, but it won't be the same, you know? Not that reliable, I suppose.

So, at last, to the question itself, all two parts of it:

1) What to do with bigrams that have never occurred in background corpus(and both unigrams are not there either, just two words we haven't seen before)

2) What to do with bigrams only one part of which has occurred in background corpus?

3) What to do with bigrams that have never occured in background corpus, but both words from it are present in background corpus?

How to make it smooth, so some scores of collocation strength can be created for all these cases and were at least a little reliable.

Thank you in advance, kick this question to whatever site is appropriate if I've misunderstood the purpose of this particular part of stackexchange.

UPD: my 9th attempt to post it ended up with trimmed question body, I'm sorry, now it's intact.

  • $\begingroup$ It looks like this got cut off as it ends oddly and without asking a question. $\endgroup$ – Peter Flom Nov 9 '12 at 11:24
  • $\begingroup$ It has in fact gotten cut off, fixed it. $\endgroup$ – Anton Nov 9 '12 at 12:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.