Computing Classification Error between HMMs Consider two Hidden Markov Models. They have different state space but have the same output space and are all left to right HMMs. Is it possible to compute in closed form the probability of a sequence generated by one say HMM1 having a higher likelihood in the other HMM (HMM2)?
What I am looking for is a closed form solution to this problem.


*

*Initialize counts for both HMMs to zeros.

*Generate a sequence from HMM1.

*Compute the likelihood of the sequence for HMM1 and HMM2. Whichever HMM has a higher likelihood increment a count.

*Do this for a large number of sequences and divide the counts by the number of sequences. This gives you the accuracy of HMM1 and also the risk HMM2 poses to HMM1 when used for classification based on likelihood.


If it helps, my final goal is to select a subset of such HMM models that optimize accuracy and risk.
If you do not know the answer to the question but have some literature that can send me searching along the right direction, please share.
 A: As the sequences are generated by HMM1 you will expect most of the data sequences to have a higher likelihood with respect to HMM1 than with respect to HMM2 as your first sentence suggests.
The experience is interesting to carry out (and is not too complicated), however, I am not sure how much the accuracy you will get out of it will be reliable from the scientific point of view (like if you plan to publish such study, you may receive a number of concerns from the reviewers).
I think you could run a second set of experiments in which data is generated by HMM1 but then a third HMM (HMM3) is trained on this data. Comparing how HMM3 behaves compared to HMM1 would help.
Or some experiments using data generated from HMM3 with likelihoods computed with respect to HMM1 and HMM2.
I played around the same sort of problematic during my PhD. It may not be the exact same problem but if this can help, you can have a look at the experimental section on synthetic data of this paper: Hidden Markov Models Based on Generalized Dirichlet Mixtures for Proportional Data Modeling, by Epaillard and Bouguila.
