Apologies for cross posting this I posted in Linguistics but so far there have been no takers.

I need to compare the frequency of occurrence of a word in two corpora.

Take for example the word fail which occurs 311 times in a corpus containing language from the sciences (2 million tokens) and 420 times in a corpus containing language from the arts (2.2 million tokens).

Many textbooks recommend the LL test for such a task. However my understanding is that: 1) the LL test assumes parametric distribution 2) that my data is not normally distributed Therefore the LL test would not be suitable for this purpose. Is this correct?

I'd really appreciate some help.


2 Answers 2


Yes, this is fine. This is called the $G$-test. It compares if $311/2000000$ and $420/2200000$ are statistically different. It is an alternative to tests like the proportion $z$-test, Fisher's exact test, and the chi-squared test.

ANOVA asks if multiple normal distributions have different means by doing a linear regression; the $G$-test asks if multiple Binomial distributions have different means by doing a logistic regression. Both compare the fit of the model considering the categorical predictor to a model that only considers the overall mean. In both tests, if the fit of the model with the categorical predictor is much better, we consider the means to be significantly different.

There are R packages that do the $G$-test for you. I find it instructive to do it yourself (at least constructing the logistic regression models yourself).

x <- c(rep(0, 2000000), rep(1, 2200000))
y <- c(rep(0, 311), rep(1, 2000000 - 311), rep(0, 420), rep(1, 2200000 - 420))
L0 <- glm(y ~ 1, family = binomial) # intercept-only model
L1 <- glm(y ~ x, family = binomial) # model with the category as the predictor
lmtest::lrtest(L0, L1) # This does the likelihood ratio test

You can use Naive Baye's classifier and adopt it as I explain below:

The classifier is based on the Bayesian Theorem which is a probabilistic theory and hence, the classifier becomes a probabilistic classifier. We can demonstrate the simple statement of the Bayesian theorem in Equation below:

$$ P(A|B) = \frac{P(B|A)P(A)}{P(B)} $$

By re-writing this Equation, we can then obtain equation below: $$ P(doc_{i}|Keyword) = \frac{P(Keyword|doc_{i})P(doc_{i})}{P(Keyword)} $$ where $doc_i$ ($1 \leq i \leq n$) has a prior probability $P(doc_i)$ , $P(doc_i|Keyword)$ is $doc_i$'s posterior probability given Keyword $Keyword$, and $P(Keyword|doc_i)$ is the conditional probability of $Keyword$ being seen in $doc_i$. This can be said that the Posterior for Keyword is equal to a fraction of likelihood multiplied by prior divided by evidence. Since in practice the evidence is a constant. This is same as if we say in Equation of Bayesian theorem for text extraction, as the denominator $P(Keyword)$ is independent of $doc_i$, and $P(doc_i)$ remains the same for all keywords, the likelihood that a search Keyword appears in $doc_i$, $P(Keyword|doc_i)$, dominates the posterior probability $P(doc_i|Keyword)$. so we just need to calculate the prior and likelihood to find the probability of $doc_i$ being chosen regarded to the Keyword. This can be done by using the Naive Bayesian Classifier. With $mean_i=\mu_i$ and $variant_i=\sigma_i$ of all the words' tf*idf values in $doc_i$ , $P(Keyword|doc_i)$ in Equation of Bayesian theorem for text extraction can thus be approximately measured through a Normal Distribution ${\cal N}(\mu_i, \sigma_i^2)$: $$ P(\text{Keyword}|\text{doc}_i)= \frac{1}{\sqrt{2\pi\sigma_i^{2}}} \times e^{\frac{-(tf(\text{doc}_i,\text{Keyword})*idf(\text{doc}_i,\text{Keyword})-\mu_i)^{2}}{2\sigma_i^{2}}} $$

To calculate the value for TFIDF, here we apply the stemmed tfidf principle. One way to calculate $tf(\text{doc}_i,\text{Keyword})*idf({doc}_i,{Keyword})$, is illustrated in the Equation below :

$$ \log (1+\frac{N_{Keyword}}{N_{total}})*\log \frac{N_{D}}{N_{KD}} $$ Where $N_{Keyword}$ is equal to the frequency of the keyword and $N_{total}$ is equal to the total frequency of all words, including the corresponding keyword, $N_D$ is equal to the total number of all documents and $N_{KD}$ is equal to the number of documents that the keyword occurrence happens there. On the other hand, $idf(doc_i,Keyword)$ is computed as the inverse of the total number of $Keyword$ appearing in all parts.

The reason for using the Normal distribution (Gaussian distribution) is for the fact that for the weighting factor for reflecting the importance of words in a page, we used the TFIDF principle which is fundamentally equal to $tf*idf$. To calculate tf, we can use the Equation above. On the other hand, this is shown in literature that the word frequency follows the Log-normal distribution. By considering the Equation above, we can see that for calculating the value of $tf(doc_i,Keyword)$, we apply the logarithm of the term frequency and hence, the conditional probability, $P(Keyword|doc_i)$, follows a normal distribution.

The above article is taken from following paper:

Mostafa Alli 
SERP-level Disambiguation from Search Results.
DOI: 10.5220/0005628606270636
In Proceedings of the 7th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management,
pages 627-636
ISBN: 978-989-758-158-8
c 2015 by SCITEPRESS - Science and Technology Publications, Lda.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.