2
$\begingroup$

So, I have seen many questions here asking whether it is a good idea to use AIC/BIC for determining the optimal number of hidden states for an HMM. What about the number of observable states though?

In my case, I am using a discrete HMM, where I quantised the continuous time-series observation signal to obtain a sequence of discrete emissions. I train a number of HMMs and then use the AIC to find the "best" one. Each HMM has a different number of hidden states (3 to 9) and a different number of values an observation can possibly be assigned after the quantisation (4 to 128).

When I use AIC, it barely makes a difference as for large number of states, the log-likelihood is too low (-2000) to compare to the punishment the AIC induces because of the free parameters. Also, the log-likelihood is always better (around -300) for low number of states.

Does after this make sense to use AIC, or should I just compare the different models (with different number of free parameters) only using the log-likelihood?

$\endgroup$
4
  • $\begingroup$ If I am getting your setup right, you cannot compare AICs as you effectively have incomparable likelihoods because you are fitting different data (different observed states). Curiously, I attended a research seminar yesterday on precisely this topic. $\endgroup$ Feb 24, 2017 at 20:15
  • $\begingroup$ I know you cannot compare likelihoods of data of different length, but I hadn't thought that if the observations belong to sets of different sizes they are also not comparable. Could you elaborate, maybe share resources from that seminar? It would be very helpful. Also, if that's the case, what could I possibly use instead of AIC? $\endgroup$
    – DimP
    Feb 25, 2017 at 3:05
  • 2
    $\begingroup$ I cannot find a research paper that would correspond to the seminar talk, but the idea is the following: if you have underlying data $x$ and later categorize it in two different ways to form $y=f(x)$ and $z=g(x)$ (both of which are based on the same $x$, but on two different coarser scales due to two different categorizations), you cannot directly compare the likelihood of $y$ and $z$, even though the underlying variable is $x$ in both cases. $\endgroup$ Feb 25, 2017 at 9:20
  • $\begingroup$ @RichardHardy thanks for your answer, I think it covers well enough my question. I cannot upvote your comments, but should be able to mark it as a accepted if you convert it to an answer. I guess then I'll have to find another way of comparing the models. $\endgroup$
    – DimP
    Feb 25, 2017 at 18:14

1 Answer 1

4
$\begingroup$

AIC is based on likelihood, and the likelihood has to be calculated for the same set of observations. If you have underlying data $x$ and later categorize it in two different ways to form $y=f(x)$ and $z=g(x)$ (both of which are based on the same $x$, but are measured on two different relatively coarse scales due to two different categorizations), then you cannot directly compare the likelihoods - nor AIC - of $y$ and $z$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.