# Likelihood based model selection

Let's say I got a set of models $M = \{M_1, M_2, \dots M_n\}$.

Now say I got some data $x$ and I would like to know, which model represents the data best.

I know how to calculate the likelihood $L(\theta | x)$, with $\theta$ being the parameters of any of those models. I realize that the likelihood value of one model alone won't tell me anything useful. But what I can do is compare them to each other.

Now I know which of the given models is the most likely. But: I would also like to know, how likely it is none of the models represents a model well enough? That is, I'm interested in a statistical sound way to tell, that I should create a new model for that data.

Any pointers on how I could calculate this?

-
In real life models are just approximations of reality. None of the models generated the data. That you can be sure of. What you can say is which model of the group gives the best fit without overfitting. –  Michael Chernick Jun 20 '12 at 23:17
You've presented a sort of abstract formulation but is there a concept of a "null" model, say, $M_0$, that reflects a believe that non of the parameters are useful? –  Macro Jun 20 '12 at 23:18
You could consult with Bayesian Model Averaging, where the prior distribution is the probability of your believe of the different models $M$. And the posterior can be calcuated together with the likelihoods from $M$. Look at the "BMS" package in R, it may have what you want. –  Fred Jun 20 '12 at 23:49
My fear based on reading the current version of the question is that some confusion may exist regarding what the (statistical) likelihood represents. The term likelihood appears to be used interchangeably in both statistical and colloquial senses. –  cardinal Jun 20 '12 at 23:56
To Michael's point, see the first quote by George E.P. Box: stats.stackexchange.com/questions/726/… –  Chap Jun 21 '12 at 1:49
The simplest way to answer the first part of your question is to direct you to information criteria like Akaike's AIC, Schwarz' BIC, and Spiegelhatler's DIC. These criteria lead to a numerical comparison of models by adding to the maximal likelihood under a given model $$\max_{\theta_i} L_i(\theta_i|\mathbf{x})$$ a penalisation term that corresponds to the "complexity" of the model. For instance, \begin{align*} AIC(M_i) &= 2k_i - 2\ln\{\max_{\theta_i} L_i(\theta_i|\mathbf{x})\}\\ BIC(M_i) &= 2k_i \ln(n) -2\ln\{\max_{\theta_i} L_i(\theta_i|\mathbf{x})\}\\ DIC(M_i) &= 2\hat k_i(\mathbf{x}) - 2 \mathbb{E}^{\theta_i}[\ln L(\mathbf{x}|\theta_i)|\mathbf{x}] \end{align*} where $DIC(M_i)$ involves a prior distribution for each model, $\pi_i(\theta_i)$, and where $\hat k_i(\mathbf{x})$ is the estimated or effective number of parameters, $$\hat k_i(\mathbf{x}) = \mathbb{E}^{\theta_i}[\ln L(\mathbf{x}|\theta_i)|\mathbf{x}] - \ln L\left(\mathbf{x}\bigg| \mathbb{E}^{\theta_i}[\theta_i |\mathbf{x}]\right)$$