1
$\begingroup$

There are several ways to build language models, n-gram based models are straightforward, but for the language models built on HMMs, what are hidden states and what are observations?

$\endgroup$
1
$\begingroup$

Generally the hidden states are the parts of speech (eg, noun, verb) and the observations are the words.

So we assume that each word (emission) depends only on the part of speech and each part of speech depends only on the part of speech preceding it in the sequence (this last one is "markov", or memoryless, assumption).

The part of speech is therefore "hidden" as it is not directly observed.

$\endgroup$
  • $\begingroup$ but as I understand it, the goal of language model is to compute the probability of a word sequence, according to what you said, we need the probability of the observation sequence, which is literally the count of the sequence occurence in the training set ... so its exactly the same as n-gram language model....why would anyone use HMM for this? $\endgroup$ – aditya sista Aug 31 '17 at 18:09
  • $\begingroup$ The probability of the observation sequence is not the count of the sequence in the dataset (the observation sequence itself does not even have to exist in the dataset); it depends now on the hidden variables. The transition matrix and probability of observations given part of speech are from counts of the observations (with pos labels) in the dataset. The idea is that the probability of a sentence does depend on its parts of speech, althouh it's meaning may not. True this right is? $\endgroup$ – user0 Aug 31 '17 at 18:23
  • $\begingroup$ okay, I have a huge text corpus that im going to train for language model (i just want to predict next word given previous words), I don't have parts of speech tags, can I use HMM for this? $\endgroup$ – aditya sista Aug 31 '17 at 18:25
  • $\begingroup$ There may be some way to do it with an HMM (you could use someone else's trained HMM), but I would recommend an LSTM. Consider the next word in the sentence "At the park, I play ___". Any model assuming markov property wouldn't know about "park", so it would just predict anything that follows play, eg, "house" or "dough" rather than "frisbee". An LSTM does remember park. If you use an HMM in the traditional sense with POS as the hidden variable, you can imagine that even with "I like to play park ___", it would not necessarily predict "frisbee" above "house" because they're both nouns. $\endgroup$ – user0 Aug 31 '17 at 18:29
  • $\begingroup$ alright, thanks for clarification, so basically you can't have HMM language model when you don't have a sensible hidden state tags. $\endgroup$ – aditya sista Aug 31 '17 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.