I am currently doing research comparing language modelling in English to language modelling in programming languages (namely Java) using perplexity as the metric for the language model being used. My question is whether different data set sizes will invalidate the comparison of the perplexities.

I am using the Penn Treebank as the English data set and it contains approximately 1,000,000 words. The other data sets I am using each contain about 200,000 to 500,000 words. All of the data sets have constant vocabulary size of 10,000.

If it matters at all, I am using a recurrent neural network. Would comparing perplexities be invalidated by the different data set sizes? Thanks.


Would comparing perplexities be invalidated by the different data set sizes?

No. I copy below some text on perplexity I wrote with some students for a natural language processing course (assume $\log$ is base 2):

In order to assess the quality of a language model, one needs to define evaluation metrics. One evaluation metric is the log-likelihood of a text, which is computed as follows, assuming that the language model is a trigram model, and the text contains $N$ words:

\begin{align} l_{\text{corpus}} = \log \left ( \prod_{i=3}^{N} p(w_i | w_{i-2},w_{i-1}) \right ) = \sum_{i=3}^N \log p(w_i | w_{i-2},w_{i-1}) \end{align}

In order to make this metric independent from the size of the corpus, one can compute the average log-likelihood of the corpus on a per word basis, i.e. the log-likehood of the corpus normalized by the number of words:

\begin{align} l_{\text{word_average}} = \frac{1}{N} \sum_{i=1}^N \log p(w_i | w_{i-2},w_{i-1}) \end{align}

The most common evaluation metric for a language model is the perplexity, which can be computed directly from the average log-likelihood of the corpus on a per word basis:

\begin{align} \text{Perplexity} \ = \ 2^{-l_{\text{word_average}}} \end{align}

Note that in general, to make a meaningful comparison between two different language models, one needs to use the same vocabulary.

Using unigram, bigram and trigram models trained on 38 million words from the Wall Street Journal, and using a vocabulary of size 19,979 one obtains a perplexity 962, 170, and 109, respectively, when tested on 1.5 million words from the same journal.

What is the intuitive meaning of perplexity? This measure can be interpreted as an actual branching factor of the model. Let's explore this intuition using a simple uniform model for unigrams: $P(w) = \frac{1}{|V|}, \forall w \in V$. This means that: \begin{align} P(w_1, ..., w_N) = \prod_{i \in { [ 1, N ] }}{P(w_i}) = \left( \frac{1}{|V|} \right)^N \notag \\ \text{perplexity} = 2^{- \frac{1}{N} P(w_1, ..., w_N)} = 2^{- \frac{1}{N} \log \left( \left( \frac{1}{|V|} \right)^N \right)} = |V| \end{align}

Under this uniform language model, the perplexity is equal to the size of the vocabulary. Generally, perplexity captures the effective vocabulary size under the model. For instance, a trigram model described above has a factual branching factor of 109, even though it operates over the vocabulary of 19,979.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.