# How to handle new observations on HMM decoding?

I'm implementing the HMM algorithms described in Rabiner's tutorial. But there is several issues to considered when we apply HMM for real problems. One of this problems is how to consider new observations that hasn't emission probabilities. Anyone knows a way of considering a distribuition for new observations? I believe that the information about previous state can be useful for inferir the state of the the new observation.

• What do you mean by "new observations"? – jerad Mar 6 '13 at 18:49
• "New observations" are new symbols. I'm working with a base which is basically product descriptions. I do not have all the possible words in my training base, so is common to appear new words (symbols) when decoding new products description. – zeferino Mar 6 '13 at 18:55

The problem you're referring to, often called data sparsity, arises often in language modeling. In particular, if some vocabulary words don't appear in your training corpus then maximum likelihood techniques will lead to a learned model that assigns $0$ probability to observing those words.
Depending on how you define an observation, you can solve this problem by have a pseudo observation for rare training observations or unseen observations, e.g. number for all numbers. That way, when the HMM encounters an unseen observation, it looks for the closest pseudo observation. See 2.7.1 in [this][1] for more details.
On the other hand, if you can not have pseudo observation in you HMM model, the simplest way to handle unseen observation is just assign them zero probabilities!