1
$\begingroup$

This article describes a hierarchical clustering algorithm which clusters the words within a vocabulary based on their similarity, in order to improve a language model (in the article, n-grams).

How and why is the language model improved by using this method?

P.S.: I see the perplexity is lower when using the clustering method, but I want to get the intuition behind this result, not just numbers.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

According to the paper, the n-gram model predicts the next word based on the previous n words. Let's get intuitive...

Say n is 4, and you have "I love to eat". That is, given "I love to eat ______", you want to fill in the blank with the most likely word.

Intuitively, I would guess that "I love to eat ____" and "I like to eat ____" would have a similar answer and thus "love" and "like" might have a similar meaning in this context. As far as I can understand it, the algorithm is trying to determine that {"love", "like"} are a word cluster.

By definition, you'll have more examples in your corpus of {"love", "like"} than you will of either "love" or "like" individually, which means your data will be less sparse when you use the cluster rather than individual words, which also helps with calculations.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Note that you can think of n-gram models as a form of clustering too. For example, a trigram model would cluster the sentences "I love to eat ___" and "I like to eat ___" together when predicting the next word because the last two words in each sentence are the same. $\endgroup$
    – Aaron
    Jul 27, 2018 at 4:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.