I am trying to fit an HMM to a dataset resembling:
id0: A, A, A, C, B, A, C, B
id1: C, B, A, A, C, C
id2: B, A, A, C, B, B, A
In total there are three characters (A, B, C) and about 40k of such rows. I'm using R's HMMFit from the RHmm package.
As input to HMMFit one has to select the desired number of hidden states, so I can generate different models for different numbers of hidden states. In addition to the model, HMMFit reports the log-likelihood, the BIC criterium and the AIC criterium.
How does one evaluate the adequacy of such a model? Should one infer the model from training data as opposed to the entire dataset like I do now? If yes, how does one check if the trained model describes the test data?
Given different potentially adequate models generated for different numbers of hidden states, which one should I prefer?
