# Should I use the Log-Likelihood (G2) test to compare word frequency across corpora?

Apologies for cross posting this I posted in Linguistics but so far there have been no takers.

I need to compare the frequency of occurrence of a word in two corpora.

Take for example the word fail which occurs 311 times in a corpus containing language from the sciences (2 million tokens) and 420 times in a corpus containing language from the arts (2.2 million tokens).

Many textbooks recommend the LL test for such a task. However my understanding is that: 1) the LL test assumes parametric distribution 2) that my data is not normally distributed Therefore the LL test would not be suitable for this purpose. Is this correct?

I'd really appreciate some help.

You can use Naive Baye's classifier and adopt it as I explain below:

The classifier is based on the Bayesian Theorem which is a probabilistic theory and hence, the classifier becomes a probabilistic classifier. We can demonstrate the simple statement of the Bayesian theorem in Equation below:

$$P(A|B) = \frac{P(B|A)P(A)}{P(B)}$$

By re-writing this Equation, we can then obtain equation below: $$P(doc_{i}|Keyword) = \frac{P(Keyword|doc_{i})P(doc_{i})}{P(Keyword)}$$ where $doc_i$ ($1 \leq i \leq n$) has a prior probability $P(doc_i)$ , $P(doc_i|Keyword)$ is $doc_i$'s posterior probability given Keyword $Keyword$, and $P(Keyword|doc_i)$ is the conditional probability of $Keyword$ being seen in $doc_i$. This can be said that the Posterior for Keyword is equal to a fraction of likelihood multiplied by prior divided by evidence. Since in practice the evidence is a constant. This is same as if we say in Equation of Bayesian theorem for text extraction, as the denominator $P(Keyword)$ is independent of $doc_i$, and $P(doc_i)$ remains the same for all keywords, the likelihood that a search Keyword appears in $doc_i$, $P(Keyword|doc_i)$, dominates the posterior probability $P(doc_i|Keyword)$. so we just need to calculate the prior and likelihood to find the probability of $doc_i$ being chosen regarded to the Keyword. This can be done by using the Naive Bayesian Classifier. With $mean_i=\mu_i$ and $variant_i=\sigma_i$ of all the words' tf*idf values in $doc_i$ , $P(Keyword|doc_i)$ in Equation of Bayesian theorem for text extraction can thus be approximately measured through a Normal Distribution ${\cal N}(\mu_i, \sigma_i^2)$: $$P(\text{Keyword}|\text{doc}_i)= \frac{1}{\sqrt{2\pi\sigma_i^{2}}} \times e^{\frac{-(tf(\text{doc}_i,\text{Keyword})*idf(\text{doc}_i,\text{Keyword})-\mu_i)^{2}}{2\sigma_i^{2}}}$$

To calculate the value for TFIDF, here we apply the stemmed tfidf principle. One way to calculate $tf(\text{doc}_i,\text{Keyword})*idf({doc}_i,{Keyword})$, is illustrated in the Equation below :

$$\log (1+\frac{N_{Keyword}}{N_{total}})*\log \frac{N_{D}}{N_{KD}}$$ Where $N_{Keyword}$ is equal to the frequency of the keyword and $N_{total}$ is equal to the total frequency of all words, including the corresponding keyword, $N_D$ is equal to the total number of all documents and $N_{KD}$ is equal to the number of documents that the keyword occurrence happens there. On the other hand, $idf(doc_i,Keyword)$ is computed as the inverse of the total number of $Keyword$ appearing in all parts.

The reason for using the Normal distribution (Gaussian distribution) is for the fact that for the weighting factor for reflecting the importance of words in a page, we used the TFIDF principle which is fundamentally equal to $tf*idf$. To calculate tf, we can use the Equation above. On the other hand, this is shown in literature that the word frequency follows the Log-normal distribution. By considering the Equation above, we can see that for calculating the value of $tf(doc_i,Keyword)$, we apply the logarithm of the term frequency and hence, the conditional probability, $P(Keyword|doc_i)$, follows a normal distribution.

The above article is taken from following paper:

Mostafa Alli
SERP-level Disambiguation from Search Results.
DOI: 10.5220/0005628606270636
In Proceedings of the 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management,
pages 627-636
ISBN: 978-989-758-158-8