5
$\begingroup$

Let's say you have a Bayesian model:

$$\theta' \sim g(\theta|\mu) $$ $$ y \sim p(y|\theta')$$

And we have some data ($n$ data points) $\mathbf{y}_n$, which we will use to perform inference on $\theta'$ by calculating $p(\theta'|\mathbf{y}_n)$

My interpretation of the posterior $p(\theta'=x|\mathbb{y}_n)$ in this case is as follows:

Given the proposed model and observations $\mathbf{y}_n$, what is the probability that the value of the generated $\theta'$ was x

Note: This is not the same as likelihood, which says "how probable is the data given $\theta'$...I am explicitly taking into account the probability of both the generated model $(\theta' = x)$ and the observations $\mathbf{y}_n$ since both are modeled as random.

Now, if the model were correct, then as I collect more data ($n \to \infty$), I'd expect the likelihood function $p(\theta'|\mathbf{y}_n)$ to concentrate more and more around a particular value of $\theta'$. Let's call this a convergent posterior.

However, most modern statisticians would probably say a model is merely a device or approximation to the true statistical model of the system under study -- this approximate model is used to provide a simplified context within which we can interpret some aspects of the system (e.g., if $\theta'$ represents some "mean response time" that we wish to model as constant for a given system...it may not be, but we want to see how much we can explain if we assume it is).

So, what about the much more likely case that our model is wrong -- what happens as we collect more data in this (highly likely) case?

I can imagine two broad scenarios:

  1. In the context of our model, the data behave as if they were generated by a single (or very small interval) of $\theta'$.
  2. The data behave as if $\theta'$ were randomly chosen for each $y_i$ from some mixture distribution over $\theta'$.

(1) is what most of us expect as we collect more data and should yield at least an approximately convergent posterior.

However, in (2) the data $\mathbf{y}_n$ appear to be generated from a mixture of $\theta'$ values, which I would expect would be shown as a non-convergent posterior for $\theta'$ (e.g., it converges to some uni-modal or multi-modal distribution).

In fact, in the extreme case, wouldn't a poorly specified model allow the possibility that the 95% posterior credible interval becomes wider than the 95% prior credible interval (since the likelihood will not "settle down" to a high-likelihood $\theta'$)?! The most pathological case would be where your model is so inconsistent with the data that all credible intervals for $\theta'$ diverge as you collect more data (i.e., you likelihood model is constantly "surprised" by new observations).

My Question:

It seems to me that from a modeling perspective, (2) should be the norm and not the exception, since the data does not actually follow our model, therefore:

What are necessary or sufficient conditions under which (2) is possible? (References to papers are also appreciated).

Here is a paper I found, which suggests my intuition about (2) is correct.

https://projecteuclid.org/download/pdf_1/euclid.aoms/1177699597 [This paper had a couple corrections published here]

Another good paper by Gelman et al: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4476974/pdf/nihms697619.pdf

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ As a comment-quality answer, what happens if the model is wrong, in general, is that the posterior converges to a point mass that is the "closest" in some well-defined sense to the true model. That point mass is the one that maximizes the probability of the data given the (wrong) model. Just because the model is wrong does not mean that there is no point in the parameter space associated with the (wrong) model that maximizes the probability of the data! $\endgroup$
    – jbowman
    Commented Apr 25, 2017 at 17:05
  • $\begingroup$ @jbowman thanks! yes, it appears that in many cases the likelihood function will converge to the point that minimizes the KL distance to the true distribution. However, in the papers I linked to at the bottom, the researcher shows that this behavior is not guaranteed. $\endgroup$
    – user145807
    Commented Apr 25, 2017 at 17:30
  • $\begingroup$ Right, when there are multiple such points - just like any optimization problem where there is no unique optimal solution. However, if you think about the MLE problems we face, we almost always have unique solutions to our maximum likelihood estimation, even though the model is always wrong - so clearly in general it is the case that we get a unique solution. And, since Bayesian posterior point estimates and MLE converge asymptotically under suitable regularity conditions, this carries over to Bayesian estimation. $\endgroup$
    – jbowman
    Commented Apr 25, 2017 at 18:00
  • $\begingroup$ @jbowman I see -- hence the importance of external model checking....the closest fit may not be that close! $\endgroup$
    – user145807
    Commented Apr 25, 2017 at 19:01
  • 1
    $\begingroup$ @jbowman your discussion appears to be relevant and useful; even if you regard it as an incomplete answer I think it would be useful to have it as an answer. Please consider posting it as one. $\endgroup$
    – Glen_b
    Commented Apr 25, 2017 at 23:08

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.