I'm currently studying Hidden Markov Models. There's a set of observations from which I need to determine the optimal number of states. After having found the maximum likelihood using Baum-Welch, I considered two model selection criteria for determining the optimal states. These are Minimum Description length (MDL) and Bayesian Inference criterion (BIC). However, with MDL, the number of states=2 whereas with BIC it's 4. Does this mean that MDL performs better than BIC?


The Bayesian Infomration Criterion (BIC) is given as:

\begin{equation}\label{eq_BIC_FINAL} BIC = \log f\left( {\bf{x}}|\hat{{\bf{\theta}}}_i ; H_i\right) - \frac{1}{2} \log \left| I\left(\hat{{\bf{\theta}}}_i \right)\right| + \frac{n_i}{2} \log 2 \pi e \overset{i}{\rightarrow} max, \end{equation}

where $i=1,\cdots,M$ is the model order index, $\left| \cdot \right|$ is the determinant, $I\left(\hat{{\bf{\theta}}}_i \right)$ is the Fisher Information Matrix for parameter ${\bf{\theta}}_i$ and $n_i$ is the number of unknown deterministic parameters under each hypothesized model.

MDL is derived directly from the BIC when $N\to \infty$ assuming i.i.d samples. Assuming $N$ i.i.d. samples we can write $I\left(\hat{\theta}_i \right) = N i\left(\hat{\theta}_i \right)$, where $i\left(\hat{\theta}_i \right)$ is the Fisher information matrix based on only one sample evaluated at $\hat{\theta}_i$. Inserting this into the BIC we get

\begin{equation}\label{eq_MDL1} \log f\left( {\bf{x}} ; H_i\right) = \log f\left( {\bf{x}} |\hat{\theta}_i ; H_i\right) - \frac{n_i}{2} \log N - \frac{1}{2} \log \left| i\left(\hat{\theta}_i \right)\right| + \frac{n_i}{2} \log 2 \pi e, \end{equation}

where $H_i$ is the hypothesized model order.

Now taking $N$ to infinity will leave only the first two terms in the above equation so we get

\begin{equation} \log f\left( {\bf{x}} ; H_i\right) = \log f\left( {\bf{x}} |\hat{\theta}_i ; H_i\right) - \frac{n_i}{2} \log N \overset{i}{\rightarrow} max. \end{equation}

Usually in the literature the signs are in the opposite direction so we wish to minimize the MDL:

\begin{equation} MDL = -\log f\left( {\bf{x}} |\hat{\theta}_i ; H_i\right) + \frac{n_i}{2} \log N \overset{i}{\rightarrow} min. \end{equation}

So, obviously if one of the assumptions made above, namely, a lot of samples and i.i.d of the samples, does not hold, MDL will not give the same results as BIC.


No, if MDL is minimized by a model with two states while BIC is minimized by a model with four, that would not of itself imply that MDL is better.

But it's possible I missed something. What would make you think so?

  • $\begingroup$ Thanks for your response. Model selection criterion are known to compress the observed data into a reduced form by using a particular model. Here's a very readable link to [Google books][1]. According to me, if MDL is giving a model which is smaller than BIC, this might reduce the computational burden for Viterbi algorithm. Your opinion and ideas will be much appreaciated. Thanks again [1]:MICAI 2006:How Good Are the Bayesian Information Criterion and the Minimum Description Length Principle for Model Selection?By Carlos Carlos Alberto Reyes $\endgroup$ – JebSam Mar 17 '13 at 4:07
  • $\begingroup$ Is your criterion for preferring MDL 'whatever gives the smallest model'? $\endgroup$ – Glen_b -Reinstate Monica Mar 17 '13 at 6:44
  • $\begingroup$ In fact I've got several observations for different types of events. There are 4 events each divided into 4 data types. BY data types I mean data smoothing, data wavelets, raw data and data smoothed wavelets. I trained all these 4 types of data individually with BIC and MDL and half of the results shows BIC = MDL and the other half shows BIC giving 2 or 3 more states than MDL. $\endgroup$ – JebSam Mar 17 '13 at 10:41
  • $\begingroup$ And this difference might be because of the BIC equation itself. BIC consists of a likelihood term and a penalty term. The penalty term depends on the amount of free parameters in the trained model for different states. And MDL does not depend on free parameters. The equation for MDL is as follows: Log L(current state) - Log L(prev state) > (N+ d^2/4 +3d/4) where L =likelihood; N= number of states and d=number of columns in the training data set (which equals to N). $\endgroup$ – JebSam Mar 17 '13 at 10:41

In a mathematical sense, there is no such thing as "better." There is only larger or smaller according to some sort of norm or other function producing real numbers and/or intervals as output.

If you ever hear anyone say something is "optimal," I recommend asking, "In what sense?" This forces them to tell how they came to that conclusion, and if it is not with a comparison between real numbers/intervals, it is probably not a scientific method.

  • $\begingroup$ Your response is more a philosophical statement about epistemology than a direct answer to the question. The OP seems to seek a description of what the MDL and BIC are measuring, and how to distinguish appropriate contexts of application for each. $\endgroup$ – Don Walpola Feb 5 at 1:57
  • $\begingroup$ Thanks @DonWalpola. You are correct, my comment was a bit snarky. MDL appears to be Bayesian in character, whereas AIC is apparently neither Bayesian nor frequentist so if you want to avoid that kerfuffle I recommend AIC. That is one good reason for using it, which I fully intend to do with vigor. $\endgroup$ – brethvoice Feb 6 at 16:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.