In 8.4.3 of the book Speech and Language Processing: An introduction to natural language processing, the two matrices transition probabilities and emission probabilities can be learned by counting as follows:

$$P(t_i|t_{i-1})=\frac{C(t_{i-1}, t_i)}{C(t_{i-1})}$$ $$P(w_i|t_i)=\frac{C(t_i, w_i)}{C(t_i)}$$ where $t_i$ is a state and $w_i$ is a word.

But I remember that it is the EM algorithm that should be applied. Am I missing something?

  • 2
    $\begingroup$ This is the fully observed case, for when you have the tag sequence (the latent variables) provided. EM a is used for when your data is “partially observed”: you don’t have the values of the latent variables. All you have is the text, not the tags. $\endgroup$ Sep 15 at 12:50
  • $\begingroup$ @AryaMcCarthy If it's unsupervised, how do we know which state represents which meaning? For instance, we only know there should be five states(A, B, C, D, and E), but we don't know which state stands for A. $\endgroup$ Sep 15 at 13:25
  • 1
    $\begingroup$ If you’ve never observed the states, then you don’t ever know. The best you can do is to try some post-hoc matching. There’s a decently famous paper about what happens to the hidden states for text by Merialdo (1994). (But remember that J&M do observe the states! That lets them steer the parameters however they want.) $\endgroup$ Sep 15 at 15:50

This is not really a HMM. They call it a HMM because it has the same structure.

  1. A base level of states ($t$) which transition from one to the next.
  2. An emitted level of states ($w$) whose observation depends on the base states.

In a true HMM, the states $t$ are hidden and you have to use EM to get a maximal likelihood estimation of what they might be.

In this case, you actually (if I've understood the text correctly) have both the $t$ and $w$ states as observations so you can compute the conditional probabilities directly by just counting.

Edit: Actually if you read a bit further in the text, it appears they transition in to considering the taggings as actual hidden states which are to be inferred (Viterbi algorithm etc.).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.