My question is regarding Gaussian Mixture models, Hidden Markov models (HMM) or any type of clustering or latent variable model, for which we can devise a likelihood function.
Specifically, I train a HMM based on a temporal dataset. Assuming I have no prior knowledge of how the dataset is distributed, I am trying to find the number of hidden states ( i.e. mixture components, latent variables, etc.) that best desribes my dataset.
My first attempt was, similar to choosing the number of clusters in a k-means setting, just use a visual elbow/knee method to spot the
#state value where the corresponding score gain (LL or AIC or Distortion measure [if I were running a k-means]) plateaues.
With LL, I can see that the score gain flattens with 8 states (although it increases again with 9 states), so I can nominate
8 as a candidate according to the elbow method. Looking at AIC, which in contrast to LL takes the model complexity into account, I guess one should go for the point where delta(AIC) goes negative rather than flattens (as in an elbow approach) because AIC is already penalising for the increase in
1) Do you agree that an elbow/knee method is not suitable for AIC due to the above reason?
2) Looking at the figure posted, what number of states would you pick?
3) Expanding on 2, and assuming one can calculate a likelihood function, what principled methods would you recommend in general for finding the optimal number of mixtures/states/latent variables in such a setting?
Any references very welcome.
EDIT: @Aleksandr Blekh has kindly provided a link to an amazing SO answer on how to choose the number of clusters for k-means, which I think covers most alternatives to likelihood-based scoring approaches. The link also covers a BIC scoring example - which I will look into.