2
$\begingroup$

So, I have seen many questions here asking whether it is a good idea to use AIC/BIC for determining the optimal number of hidden states for an HMM. What about the number of observable states though?

In my case, I am using a discrete HMM, where I quantised the continuous time-series observation signal to obtain a sequence of discrete emissions. I train a number of HMMs and then use the AIC to find the "best" one. Each HMM has a different number of hidden states (3 to 9) and a different number of values an observation can possibly be assigned after the quantisation (4 to 128).

When I use AIC, it barely makes a difference as for large number of states, the log-likelihood is too low (-2000) to compare to the punishment the AIC induces because of the free parameters. Also, the log-likelihood is always better (around -300) for low number of states.

Does after this make sense to use AIC, or should I just compare the different models (with different number of free parameters) only using the log-likelihood?

$\endgroup$
  • $\begingroup$ If I am getting your setup right, you cannot compare AICs as you effectively have incomparable likelihoods because you are fitting different data (different observed states). Curiously, I attended a research seminar yesterday on precisely this topic. $\endgroup$ – Richard Hardy Feb 24 '17 at 20:15
  • $\begingroup$ I know you cannot compare likelihoods of data of different length, but I hadn't thought that if the observations belong to sets of different sizes they are also not comparable. Could you elaborate, maybe share resources from that seminar? It would be very helpful. Also, if that's the case, what could I possibly use instead of AIC? $\endgroup$ – DimP Feb 25 '17 at 3:05
  • 2
    $\begingroup$ I cannot find a research paper that would correspond to the seminar talk, but the idea is the following: if you have underlying data $x$ and later categorize it in two different ways to form $y=f(x)$ and $z=g(x)$ (both of which are based on the same $x$, but on two different coarser scales due to two different categorizations), you cannot directly compare the likelihood of $y$ and $z$, even though the underlying variable is $x$ in both cases. $\endgroup$ – Richard Hardy Feb 25 '17 at 9:20
  • $\begingroup$ @RichardHardy thanks for your answer, I think it covers well enough my question. I cannot upvote your comments, but should be able to mark it as a accepted if you convert it to an answer. I guess then I'll have to find another way of comparing the models. $\endgroup$ – DimP Feb 25 '17 at 18:14
4
$\begingroup$

AIC is based on likelihood, and the likelihood has to be calculated for the same set of observations. If you have underlying data $x$ and later categorize it in two different ways to form $y=f(x)$ and $z=g(x)$ (both of which are based on the same $x$, but are measured on two different relatively coarse scales due to two different categorizations), then you cannot directly compare the likelihoods - nor AIC - of $y$ and $z$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.