3
$\begingroup$

I have a (very) small corpus of documents. As a representative example: 450 documents, 280000 total word count.

I am calculating Positive Pointwise Mutual Information (PPMI) between a selection of keyword pairs to investigate the patterns of co-occurrence between these (pre-defined) keywords.

My window for co-occurrence is a whole document, so I have 450 windows.

Here is the PPMI "standard" formula:

\begin{equation} \operatorname{PPMI}(w_1, w_2) = \max \left\{ \log_2 \left(\frac{P(w_1, w_2)}{P(w_1) \cdot P(w_2)}\right), 0 \right\} \end{equation}

There are however different ways to calculate the probability of co-occurrence P(w1,w2) and the individual probabilities of keyword occurrence P(w1) and P(w2).

I'd be keen to understand which one is most suitable to my case (small corpus and some very low individual keyword frequencies).

As an example, the individual frequencies and co-occurrence in my data for three keywords:

  kword1 kword2 count_kw1 count_kw2 count_cooccurrence_kw1kw2
1      a      b       331       131                        20
2      a      c       331         2                         2
3      b      c       131         2                         0

Method 1

Reflects the approach discussed in Church and Hanks (1990) to divide both the count of individual word frequencies and co-occurrences by the corpus length N, where N = 280000 in my case.

Results:

  kword1 kword2     P_kw1     P_kw2     P_cooccurrence_kw1kw2     ppmi
1      a      b  0.001182  0.000468                   7.1e-05   7.0039
2      a      c  0.001182  0.000007                   7.0e-06   9.7246
3      b      c  0.000468  0.000007                   0.0e+00       NA

Issues:

  • The maximum possible co-occurrence value, given that my window is the whole document, is equal to the number of documents in the corpus, i.e., 450. But to calculate the joint probability of two keywords, the count of co-occurrence is divided by corpus length. I am afraid I am underestimating the joint probabilities this way.
  • PPMI is inflated when one of the two keywords has very low individual frequency.

Method2

Terra and Clark (2003) report two different methods: the first one differs from Method1 by dividing the count of co-occurrences count(w1, w2) by the total number of windows in the corpus (instead of total number of words).

This seems to address the first issue above but not the second. Note that I tried Laplace add-1 smoothing but that did not address the impact of rare keywords on PPMI.

Results:

  kword1 kword2     P_kw1     P_kw2     P_cooccurrence_kw1kw2     ppmi
1      a      b  0.001182  0.000468                    0.0444   16.294
2      a      c  0.001182  0.000007                    0.0044   19.006
3      b      c  0.000468  0.000007                   0.0e+00       NA

Method3 Other method from Terra and Clark (2003):

  • P(w) = word frequency as document frequency (the number of documents in which the word occurs) divided by total number of documents.
  • P(w1, w2) = the number of documents where the two words co-occur divided by total number of documents (as for Method2).

Results:

  kword1 kword2 count_kw1 count_kw2 count_kw1kw2  P_kw1  P_kw2  P_kw1kw2   ppmi
1      a      b       253        62           20  0.562  0.138    0.0444      0
2      a      c       253         2            2  0.562  0.004    0.0044      0 
3      b      c        62         2            0  0.138  0.004   0.0e+00     NA

So, very very different results.

I'd appreciate any thought/advice on what methods is most suitable for such a small corpus and low raw frequencies.

Improve this question
$\endgroup$

0

Reset to default