I was reading recently about Minimum Description Length (MDL) that does not have to assume that the true model is in the model class. When this feature of MDL is highlighted in the literature, the standard statistical inference is usually criticised at the same time that its methods are designed under the assumption that they contain the true distribution. The pioneering papers of Rissanen and others mention this.

My question is: Which of the classic statistical methods make the assumption that they have the true model in the model class or that the true model has to be considered in the analysis? What happens to the results if the true model is not in the model class (which will be the case most of the time, I believe)? Are the results still valid in such cases?

I can clearly see why MDL does not assume that it has access to the true model, but it is not clear to me why the opposite is true in more traditional statistical models.

  • 4
    $\begingroup$ "...the assumption that they have the true model in the model class..."---this is assumed in all of classical (parametric) statistics. E.g. take any foundational result---just to name a few, Neyman-Pearson, existence of most powerful or minimax test, Maximum Likelihood Principle, etc. Some results, however, can be extended to the setting where the true model does not lie in the class of models being considered, i.e. under mis-specification. For example, the maximum likelihood estimator minimizes KL divergence between the class over which likelihood is maximized and the true model. $\endgroup$ – Michael Sep 4 '20 at 21:49
  • $\begingroup$ MDL drops the whole concept of "true model". $\endgroup$ – carlo Sep 5 '20 at 1:22

Both classical and Bayesian parametric methods will assume a "model form" that manifests in a likelihood function parameterised by some unknown parameters. Both assume the structure of the likelihood function, so they make the same structural "model" assumptions. Unlike Bayesian statistics, classical statistics treats the parameters as "unknown constants". This methodology proceeds by looking at statistical properties that are conditional on the true parameter values ---e.g., bias, consistency, MSE, etc. This analysis does not assume that the true parameters (and therefore the true distribution) is accessible. Instead, the classical methodology analyses statistical properties over the class of possible values of the parameters, and tries to formulate methods that work well regardless of the true value of the underlying parameter. In this sense, the methods used in classical statistics tend to be quite robust, at least within the assumed model form.

Perhaps an example will assist here. Suppose you have a parametric model for data $\mathbf{x}$ that depends on an unknown parameter $\theta \in \Theta$. One of the concepts formed in classical statistics is the idea of "consistency" of an estimator. Broadly speaking, for an estimator $\hat{\theta}$ of the parameter $\theta$, this means that $\hat{\theta} \rightarrow \theta$ (in some appropriate probabilistic sense) as $n \rightarrow \infty$. In order for the estimator to be considered "consistent", this property must hold for all $\theta \in \Theta$. Thus, so long as the assumed model form for the likelihood function is correct, a consistent estimator will tend to estimate well in the long run --- this is guaranteed regardless of the unknown parameter value. (Note here that the theoretical derivation of the consistency property involves looking at the behaviour of an estimator under an assumed true parameter value, and varying the assumed true value over the entire parameter space. However, once a particular estimator is known to be consistent within a particular model, for applied work we do not assume that the true parameter is accessible.)

The above discussion is focused on the case where the assumed model form is correct. Both Bayesian and classical statistics also have sub-fields where they look at what happens when the assumed model form is wrong, and this is the field of "mis-specification". Classical statistics has a number of well-known results for estimators in mis-specified models (see e.g., a related question here). Generally speaking, all discussions of parametric models assume that the specified parametric model is correct, and if you want to know what happens when it is not correct you go further out into the field of mis-specification. Obviously it is desirable to look at both of these issues, and it is useful when you have methods that work well for a parametric model that you think is correct, but are also robust to mis-specification.

As to your specific questions, these are all extremely broad questions that are essentially asking for a full account of the field of mis-specified models. The answer to your questions depends on which specific models you are talking about. There is a large statistical literature on mis-specification, and this has been examined for many parametric models. The "validity" of parametric methods under mis-specification varies depending on the original parametric model, the estimation method used, and the degree to which the true model departs from the originally assumed form. If you would like to know more about this, I suggest you search the statistical literature on "mis-specification". Some important papers to get you started are White (1982), McGuirk, Driscoll and Alwang (1993), Gustafson (2001) and Mayo and Spanos (2004).

  • $\begingroup$ I think this misses the point of the question, which is more about the concept of "data generating process". MDL too needs some information about the model form. $\endgroup$ – carlo Sep 5 '20 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.