3
$\begingroup$

I am testing a HMM model by generating data from a 3x3 transition matrix and 3x4 emission matrix and then trying to train a HMM model against this data with different initializations. When I plot the log-likelihood of the observations given the model for different dimensions of the transition matrix/emission matrix I have found that the log-likelihood increases with the number of states. However, I would have expected the log-likelihood to peak at 3 states, since this is the number of states that generated the data, but this does not seem to be the case. What could cause this behaviour?

Thank you in advance!

$\endgroup$

1 Answer 1

2
$\begingroup$

This is a well-known phenomenon called overfitting (see the Wikipedia article). Here is the intuition : the complexity of your model is going to increase with the number of states. And a more complicated model is able to explain more data than a simple one : intuitively, a simple model can only represent simple distributions, while a more complicated model can account for more complex phenomenons. So the likelihood can only increase with the complexity of the model.

But selecting a model solely based on its likelihood (i.e. choosing the more complicated model) is a bad idea, because such a model would poorly generalize. It would not be able to represent new data points that were not seen in the training set, while a simpler model could.

To compare your models, you have to:

  • Either split your observations into a training set (on which your models will be trained) and a test set (on which the likelihoods of different models can be compared fairly). This procedure is called cross validation (check the name of this website !) I recommend to have a look at Andrew Ng notes on overfitting and regularization.
  • Or regularize your criterion : you need to add to your likelihood a penalty term for the complexity of the model. For instance, in the Bayesian Information Criterion, a penalty term that depends on the number of free parameters in your model is added.
$\endgroup$
2
  • 1
    $\begingroup$ Thank you for your answer! I am familiar with overfitting, however I thought the case was different with HMMs since a HMM usually is representing a system in which there is natural number of states and thus I thought that there would be a optimal number of states corresponding to the system we are modelling. $\endgroup$
    – lajisam
    Commented Oct 15, 2020 at 12:44
  • $\begingroup$ No, that is actually similar. The complexity of your model depends on the number of possible hidden states. Imagine a degenerate case where you have as many states as observations. Each state would exactly correspond to one observation, and your likelihood would be very high, but your model would clearly overfit. You can understand a hidden state as a component in a mixture of distributions : the more possible hidden states you have, the more complicated your model is, which leads to overfitting. $\endgroup$ Commented Oct 15, 2020 at 12:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.