Likelihood based model selection Let's say I got a set of models $M = \{M_1, M_2, \dots M_n\}$. 
Now say I got some data $x$ and I would like to know, which model represents the data best.
I know how to calculate the likelihood $L(\theta | x)$, with $\theta$ being the parameters of any of those models. I realize that the likelihood value of one model alone won't tell me anything useful. But what I can do is compare them to each other.
Now I know which of the given models is the most likely.
But: I would also like to know, how likely it is none of the models represents a model well enough? That is, I'm interested in a statistical sound way to tell, that I should create a new model for that data.
Any pointers on how I could calculate this?
 A: First, if your models are not embedded, i.e., if one is not a simplification/complexification of another, you cannot directly use the likelihood ratio test.
The simplest way to answer the first part of your question is to direct you to information criteria like Akaike's AIC, Schwarz' BIC, and Spiegelhatler's DIC. These criteria lead to a numerical comparison of models by adding to the maximal likelihood under a given model
$$
\max_{\theta_i} L_i(\theta_i|\mathbf{x})
$$
a penalisation term that corresponds to the "complexity" of the model. For instance,
\begin{align*}
AIC(M_i) &= 2k_i - 2\ln\{\max_{\theta_i} L_i(\theta_i|\mathbf{x})\}\\
BIC(M_i) &= 2k_i \ln(n) -2\ln\{\max_{\theta_i} L_i(\theta_i|\mathbf{x})\}\\
DIC(M_i) &= 2\hat k_i(\mathbf{x}) - 2 \mathbb{E}^{\theta_i}[\ln L(\mathbf{x}|\theta_i)|\mathbf{x}]
\end{align*}
where $DIC(M_i)$ involves a prior distribution for each model, $\pi_i(\theta_i)$, and where $\hat k_i(\mathbf{x})$ is the estimated or effective number of parameters,
$$
\hat k_i(\mathbf{x}) = \mathbb{E}^{\theta_i}[\ln L(\mathbf{x}|\theta_i)|\mathbf{x}] - \ln L\left(\mathbf{x}\bigg| \mathbb{E}^{\theta_i}[\theta_i |\mathbf{x}]\right)
$$
As a Bayesian,and from a philosophical viewpoint as well, I do not have an answer to your second question! We need models (including nonparametric ones) to run the comparison. As Sherlock pounded on Watson, "when you have eliminated the impossible, whatever remains, however improbable, must be the truth"
