3
$\begingroup$

Let's say I got a set of models $M = \{M_1, M_2, \dots M_n\}$.

Now say I got some data $x$ and I would like to know, which model represents the data best.

I know how to calculate the likelihood $L(\theta | x)$, with $\theta$ being the parameters of any of those models. I realize that the likelihood value of one model alone won't tell me anything useful. But what I can do is compare them to each other.

Now I know which of the given models is the most likely. But: I would also like to know, how likely it is none of the models represents a model well enough? That is, I'm interested in a statistical sound way to tell, that I should create a new model for that data.

Any pointers on how I could calculate this?

Improve this question
$\endgroup$
8
  • 3
    $\begingroup$ In real life models are just approximations of reality. None of the models generated the data. That you can be sure of. What you can say is which model of the group gives the best fit without overfitting. $\endgroup$
    – Michael R. Chernick
    Commented Jun 20, 2012 at 23:17
  • $\begingroup$ You've presented a sort of abstract formulation but is there a concept of a "null" model, say, $M_0$, that reflects a believe that non of the parameters are useful? $\endgroup$
    – Macro
    Commented Jun 20, 2012 at 23:18
  • 1
    $\begingroup$ You could consult with Bayesian Model Averaging, where the prior distribution is the probability of your believe of the different models $M$. And the posterior can be calcuated together with the likelihoods from $M$. Look at the "BMS" package in R, it may have what you want. $\endgroup$
    – Fred
    Commented Jun 20, 2012 at 23:49
  • 1
    $\begingroup$ My fear based on reading the current version of the question is that some confusion may exist regarding what the (statistical) likelihood represents. The term likelihood appears to be used interchangeably in both statistical and colloquial senses. $\endgroup$
    – cardinal
    Commented Jun 20, 2012 at 23:56
  • 2
    $\begingroup$ To Michael's point, see the first quote by George E.P. Box: stats.stackexchange.com/questions/726/… $\endgroup$
    – Chap
    Commented Jun 21, 2012 at 1:49

1 Answer 1

Reset to default
6
$\begingroup$

First, if your models are not embedded, i.e., if one is not a simplification/complexification of another, you cannot directly use the likelihood ratio test.

The simplest way to answer the first part of your question is to direct you to information criteria like Akaike's AIC, Schwarz' BIC, and Spiegelhatler's DIC. These criteria lead to a numerical comparison of models by adding to the maximal likelihood under a given model $$ \max_{\theta_i} L_i(\theta_i|\mathbf{x}) $$ a penalisation term that corresponds to the "complexity" of the model. For instance, \begin{align*} AIC(M_i) &= 2k_i - 2\ln\{\max_{\theta_i} L_i(\theta_i|\mathbf{x})\}\\ BIC(M_i) &= 2k_i \ln(n) -2\ln\{\max_{\theta_i} L_i(\theta_i|\mathbf{x})\}\\ DIC(M_i) &= 2\hat k_i(\mathbf{x}) - 2 \mathbb{E}^{\theta_i}[\ln L(\mathbf{x}|\theta_i)|\mathbf{x}] \end{align*} where $DIC(M_i)$ involves a prior distribution for each model, $\pi_i(\theta_i)$, and where $\hat k_i(\mathbf{x})$ is the estimated or effective number of parameters, $$ \hat k_i(\mathbf{x}) = \mathbb{E}^{\theta_i}[\ln L(\mathbf{x}|\theta_i)|\mathbf{x}] - \ln L\left(\mathbf{x}\bigg| \mathbb{E}^{\theta_i}[\theta_i |\mathbf{x}]\right) $$

As a Bayesian,and from a philosophical viewpoint as well, I do not have an answer to your second question! We need models (including nonparametric ones) to run the comparison. As Sherlock pounded on Watson, "when you have eliminated the impossible, whatever remains, however improbable, must be the truth"

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ +1. Regarding the second question, how about a "goodness of fit test" (Classical or Bayesian)? This is the closest statistical concept to the OP's question I found in the sense that even if we select a model, a goodness of fit test might detect a poor fit. $\endgroup$
    – user10525
    Commented Jun 21, 2012 at 7:34
  • $\begingroup$ I think "goodness of fit" is a great keyword here :D What test would you use in a Gaussian Processes setting? $\endgroup$
    – Tobias Domhan
    Commented Jun 21, 2012 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.