I'm implementing the HMM algorithms described in Rabiner's tutorial. But there is several issues to considered when we apply HMM for real problems. One of this problems is how to consider new observations that hasn't emission probabilities. Anyone knows a way of considering a distribuition for new observations? I believe that the information about previous state can be useful for inferir the state of the the new observation.
$\begingroup$
$\endgroup$
2
-
$\begingroup$ What do you mean by "new observations"? $\endgroup$– jeradCommented Mar 6, 2013 at 18:49
-
$\begingroup$ "New observations" are new symbols. I'm working with a base which is basically product descriptions. I do not have all the possible words in my training base, so is common to appear new words (symbols) when decoding new products description. $\endgroup$– zeferinoCommented Mar 6, 2013 at 18:55
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
The problem you're referring to, often called data sparsity, arises often in language modeling. In particular, if some vocabulary words don't appear in your training corpus then maximum likelihood techniques will lead to a learned model that assigns $0$ probability to observing those words.
In language modeling, some type of smoothing method is used on the paramater estimates to solve this problem. One option, perhaps the most principled one, is to put Dirichlet priors on the probability vectors you wish to infer and perform Bayesian inference. Loosely speaking, this essentially has the effect of adding some "pseudo observations" of every vocabulary item to your training set.
$\begingroup$
$\endgroup$
Depending on how you define an observation, you can solve this problem by have a pseudo observation for rare training observations or unseen observations, e.g. number for all numbers. That way, when the HMM encounters an unseen observation, it looks for the closest pseudo observation. See 2.7.1 in [this][1] for more details.
On the other hand, if you can not have pseudo observation in you HMM model, the simplest way to handle unseen observation is just assign them zero probabilities!
