2 of 2
deleted 2 characters in body; edited title
user avatar
user avatar

Calculating pointwise mutual information between two strings

I have a dataset consisting 5000 sentences. I need to calculate PMI between 3-gram and 5-grams in this dataset.

For example:

The 5-gram is: $x_1$$x_2$$x_3$$x_4$$x_5$

And the 3-gram is: $x_2$$x_3$$x_4$

How can I calculate PMI($x_2$$x_3$$x_4$, $x_1$$x_2$$x_3$$x_4$$x_5$) in this dataset? What is the exact formula?

As far as I know, in order to calculate PMI(y,z), it is needed to keep track of these counts in the dataset:

Count(y,z) --> The number of co-occurrences of y and z

Count(i,z) --> The number of occurrences of z

Count(y,i) --> The number of occurrences of y

N --> The sample size

The final formula is:

$PMI(y,z) = \frac{Count(y,z)N}{Count(i,z)Count(y,i)}$

In the context of my problem, these counts are listed as below:

$Count_1$($x_2$$x_3$$x_4$, $x_1$$x_2$$x_3$$x_4$$x_5$) : All occurrences of exact 5-gram " $x_1$$x_2$$x_3$$x_4$$x_5$ "

$Count_2$(i, $x_1$$x_2$$x_3$$x_4$$x_5$) : All occurrences of exact 5-gram "$x_1$$x_2$$x_3$$x_4$$x_5$ " !!!!!

$Count_3$($x_2$$x_3$$x_4$, i) : All occurences of 5-grams in the form of " _ $x_2$$x_3$$x_4$ _ ", that is the first and last words are subsituted with all the possible words in the dataset.

$N$ : All the 3-grams. !!!!!

My problem is with $Count_2$ and $N$. As you see, $Count_2$ is equal to $Count_1$. Is it sensible? And I'm not sure about the way I have counted $N$.

Moh
  • 499
  • 1
  • 4
  • 16