I am in the process of building a question answering system. I am interested in calculating the PMI for words $x$ and $y$ occurring within 5 words of each other in a document. I have the formula and code but would appreciate any insight to see if I am calculating correctly.

Say I have the document (with removed stopwords):

document = 'I creating code python calculates point mutual information document python code coming along nicely'

The 5-gram (word level) output is:

[('I', 'creating', 'code', 'python', 'calculates'),
 ('creating', 'code', 'python', 'calculates', 'point'),
 ('code', 'python', 'calculates', 'point', 'mutual'),
 ('python', 'calculates', 'point', 'mutual', 'information'),
 ('calculates', 'point', 'mutual', 'information', 'document'),
 ('point', 'mutual', 'information', 'document', 'python'),
 ('mutual', 'information', 'document', 'python', 'code'),
 ('information', 'document', 'python', 'code', 'coming'),
 ('document', 'python', 'code', 'coming', 'along'),
 ('python', 'code', 'coming', 'along', 'nicely')]

Say the words I am interested in for a PMI score are 'python' and 'code'. Then the PMI would be: $$P(x,y) = \dfrac{C(x \text{ and } y)_{5G}}{N}$$

$$P(x) = \dfrac{C(x)}{N}$$

$$P(y) = \dfrac{C(y)}{N}$$

where $N$ is the number of words in the document.

The final formula for PMI would be:

$$PMI(x,y)_{5G} = \dfrac{N*C(x \text{ and } y)_{5G}}{C(x)C(y)}$$

Looking through the calculated 5-grams we see that 'python' and 'code' appear in 7 of the 5-grams. The word 'python' appears 2 times in the document. The word 'code' appears 2 times in the document.

$$PMI(\text{python},\text{code}) = \log(\dfrac{14*7}{2*2}) = 3.1986731175506815$$

Is this the correct way to calculate the PMI for words in a n-gram in a document?

Also will the calculation hold if I increase or decrease n in the n-gram?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.