Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I want to use BIC for HMM model selection:

BIC = -2*logLike + num_of_params * log(num_of_data)

So how do I count the number of parameters in the HMM model. Consider a simple 2-state HMM, where we have the following data:

data = [1 2 1 1 2 2 2 1 2 3 3 2 3 2 1 2 2 3 4 5 5 3 3 2 6 6 5 6 4 3 4 4 4 4 4 4 3 3 2 2];
model = hmmFit(data, 2, 'discrete');
model.pi = 0.6661    0.3339;
model.A = 
    0.8849    0.1151
    0.1201    0.8799
model.emission.T = 
    0.2355    0.5232    0.2259    0.0052    0.0049    0.0053
    0.0053    0.0449    0.2204    0.4135    0.1582    0.1578
logLike = hmmLogprob(model,data);
logLike =  -55.8382

So I think:

Nparams = size(model.A,2)*(size(model.A,2)-1) + 
          size(model.pi,2)-1) + 
          size(model.emission.T,1)*(size(model.emission.T,2)-1)
Nparams = 13

So at the end we have:

BIC = -2*logLike + num_of_params*log(length(x))
BIC = 159.6319

I've found a solution where the formula for num_of_params (for simple Markov model) looks like:

Nparams = Num_of_states*(Num_of_States-1) - Nbzeros_in_transition_matrix

So what's the right solution? Do I have to take into account some zero probabilities in transition or emission matrices?

====Updated since 07.15.2011====

I think I can provide some clarification on the impact of data dimension (using “Gaussian mixture distribution” example)

X is an n-by-d matrix where (n-rows correspond to observations; d-columns correspond to variables (Ndimensions).

X=[3,17 3,43
   1,69 2,94
   3,92 5,04
   1,65 1,79
   1,59 3,92
   2,53 3,73
   2,26 3,60
   3,87 5,01
   3,71 4,83
   1,89 3,30 ];
[n d] = size(X); 
n = 10; d =2;

The model will have the following number of parameters for GMM:

nParam = (k_mixtures – 1) + (k_mixtures * NDimensions ) + k_mixtures * Ndimensions  %for daigonal covariance matrices
nParam = (k_mixtures – 1) + (k_mixtures * NDimensions ) + k_mixtures * NDimensions * (NDimensions+1)/2; %for full covariance matrices

If we treat X as 1-dimentional data, than we have num_of_data = (n*d), so for the 2-dimentional data we have num_of_data = n.

2-dimentional data: nParam = 11 ; logLike = -11.8197; BIC = 1.689

1-dimentional data: nParam = 5 ; logLike = -24.8753; BIC = -34.7720

I have a very little practice with HMM. Is it normal to have HMM with (5000, 6000 and more parameters)?

share|improve this question
    
do you have a justification for using BIC? It can give horribly wrong results if not with the appropriate assumptions. –  suncoolsu Jul 15 '11 at 12:07
    
@suncoolsu , What do you mean about justification? I've found some examples on K-clusters (GMM models) selection based on BIC scoring. Probably I've provided wrong example with comparing two models with different input data (dimensions). –  Sergey Jul 18 '11 at 8:41
1  
I meant using BIC only if the assumption, the true model is in the model space, is justified. May be it is justified in your case. I agree with you that people use BIC like AIC, but both are very different things! –  suncoolsu Jul 18 '11 at 8:44
add comment

2 Answers 2

up vote 5 down vote accepted

The question is whether some of your parameters in the transition matrix and / or emission matrix are fixed to begin with. Your computations (of the number of parameters) look correct. If you for some reason want a 3 state model instead of a 2 state model and decide upfront that transitions from state 1 to 3 and 3 to 1 are not allowed (have 0 probability) you would have to take this into account in the computation of the number of parameters.

share|improve this answer
    
Do I have to take data dimension into account? What if size(data) will be 2x100 –  Sergey Jul 12 '11 at 4:52
    
@Sergey, I only looked at the computation of the number of parameters in my answer. For completeness I should have considered how to compute the "size" of the data set too, as you are interested in BIC. To be honest, I have only seen a derivation of BIC for independent data, but my guess is that the correct "size" of the data is the length of the data series. I am not sure what you mean by the size being 2x100. If you want to comment on that, you have to give a more precise description of how the data are encoded as a 2 by 100 matrix. –  NRH Jul 14 '11 at 7:55
    
Thanks for the answer, I think I understand the impact of the data dimension on BIC, please look at my update –  Sergey Jul 15 '11 at 8:10
    
Do I have to decrease number of parameters in left-right (Bakis) HMM model (with triangular Transition matrix)?Is it normal to have HMM with (5000, 6000 and more parameters)? –  Sergey Jul 18 '11 at 9:31
add comment

When we're calculating the number of free parameters in the model selection BIC, this means it's simply the number of zeros in the transtion and emission matrices. For example when there's a zero in the transition matrix-this means that there's no probability that a certain state moves to the next (as defined by the transition matrix). That's how BIC selects the optimum of states for an HMM. However, obtaining the no of free parameters just using the size of the intial, transition and emission matrices is confusing

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.