1
$\begingroup$

I'm experimenting with an HMM. I have a sequence of observations (10000) and the original matrices A,B and pi that generated those observations. There are 4 types of observations.

What I am trying to do is to train a randomly initialized HMM (initial distributions are near uniform) with different numbers of states in order to see its behavior for numbers of states lower or higher than the one that generated them.

The original model that generated the observations had 3 states.

I observed that for a number 7 or more states a lot of the matrices' values tend to zero. Is there some explanation for the reason why this would happen?

I'm not trying to find why it is happening specifically in my case. I'm not trying to solve a problem of mine.

I'm in fact curious to find whether this is something that happens in general and there is some explanation behind this.

Thank you.

$\endgroup$
2
$\begingroup$

I assume you have limited sequential data, and want to try a very complicated HMM.

Suppose you have transition matrix $A$ is $7 \times 7$ and emission matrix $B$ is $7 \times 4$ (assume you have 4 different types of observations), that is $7+7\times7+7\times4=84$ parameters, to fit such a model, it is better to have a sequence with thousands of observations.

You may experience a over-fitting problem in HMM.

HMM is similar to mixture of Gaussian, that if you increase number of hidden states, the fitting will always be better (in terms of the likelihood to be optimized).

As a result, the fitting algorithm will stop easily and find a good enough solution for the data given. I suspect this is why you are seeing most zeros in 7 hidden state transition matrix.

$\endgroup$
  • $\begingroup$ I have 4 different type of observations and have used up to 10000 observations in the training observation sequence. $\endgroup$ – MattSt Sep 19 '17 at 20:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.