Why model order selection is a big problem in statistics?

Question

I’m learning statistical signal processing for my studies. I was doing a bit of literature review on model order selection and I didn’t want to miss out on techniques that I might not have seen. I have come across some information criterion like AIC and BIC. Here, although I understood what they are, I feel without the knowledge of parameters themselves (which again depend on model order and are unknown ), these criteria are not best suited to find the model order directly. Correct me if I’m wrong. The second approach that I found was popular was the reversible jump MCMC. They try to jointly estimate model order and the parameters. In my opinion, it’s not very efficient because there are no “intuitive” jump strategies. However, I understand why it’s needed to have some idea about the parameters simultaneously when estimating the model order. The third type of estimators I saw was some sort of matrix sub space based methods where they don’t use any information about the parameters at all; like Subspace based automatic model order selection (SAMOS). I do not completely understand how they work, because they don’t use the information of parameters at all in the estimation of model order? Do they already use a lot of samples of the data so that they have enough resolution already?

Being a newbie in statistics, all I can think about model order selection is that it’s like finding the number of poles of the system. Are there any closed form calculus based methods where they estimate the number of poles (without knowing the location of the poles)? In my mind,it doesn’t intuitively make sense. Is there a comparative study or basic understanding of model order selection based on finding number of poles?

Answer 1

Model selection criteria (such as the AIC and the BIC) can perfectly be used to determine the order of a model. There is a large array of literature using them to determine the order of a Hidden Markov Model or of an autoregressive process, see for instance:

Hannan, E. J., & Quinn, B. G. (1979). The determination of the order of an autoregression. Journal of the Royal Statistical Society: Series B (Methodological), 41(2), 190-195.

Dridi, N., & Hadzagic, M. (2018). Akaike and Bayesian information criteria for hidden Markov models. IEEE Signal processing letters, 26(2), 302-306.

You said that "without the knowledge of parameters themselves [...] these criteria are not best suited to find the model order directly". Well, inferring the parameters is precisely a necessary step when using these criteria. If you fit a model $\mathcal{M}_p$ of order $p$ on your data $\mathcal{D}$, its BIC will be

$$ -2 \log p(\mathcal{D}|\hat{\theta}_p,\mathcal{M}_p) + |\mathcal{D}|k_p $$

where $\hat{\theta}_p$ is the MLE

$$ \hat{\theta}_p = argmax_{\theta} \ p(\mathcal{D}|\theta,\mathcal{M}_p) $$

Using these model selection criteria thus first implies to estimate the parameters for a given order $p$. To compute what is the optimal order $p$ (according to your criterion), you will need to compute the AIC/BIC of $\mathcal{M}_p$ (for different values of $p$), and determine which value of $p$ gives you the best fit of your data.

You can also refer to these previous questions: Number of states in HMM and Akaike Information Criterion I cannot interpret the result.

Regarding reversible jump MCMC, I agree that different strategies have pros and cons, the following paper might be useful for you:

Marrs, A. (1997). An application of reversible-jump MCMC to multivariate spherical Gaussian mixtures. Advances in neural information processing systems, 10.

Finally, subspace-based methods estimate the order of a model using a (possibly arbitrary) criterion on the eigenvalues of the Hankel matrix of the observations. This is nicely explained in the BRML textbook (Chapter 24.5.3) accessible online and which is a must-read if you are new to statistics.

Another textbook (freely accessible) that I highly recommend is MacKay, D. J., & Mac Kay, D. J. (2003). Information theory, inference and learning algorithms. Cambridge university press.