I think this quote perhaps highlights the main issue (Gelman et al. 2014; chapter 7, section 7.5. pg. 184):
In general, the posterior distribution of the model parameters can either overestimate or
underestimate different aspects of ‘true’ posterior uncertainty. The posterior distribution
typically overestimates uncertainty in the sense that one does not, in general, include all of
one’s substantive knowledge in the model; hence the utility of checking the model against
one’s substantive knowledge. On the other hand, the posterior distribution underestimates
uncertainty in two senses: first, the assumed model is almost certainly wrong—hence the
need for posterior model checking against the observed data—and second, other reasonable
models could have fit the observed data equally well, hence the need for sensitivity analysis.
The key message is that 'every model is wrong', i.e. our statistical models are just abstractions of the real world, and we should not assume that they are exact representations of the world. For instance, Richard McElreath in his book Statistical Rethinking (an excellent book on Bayesian analysis, and more accessible that Gelman et al. IMO) refers to statistical models as 'statistical golems' that spit out results of our analyses.
Therefore, given a posterior distribution, Gelman et al. (2014) are discussing ways of checking just how wrong are models are.
- Posterior distributions will likely overestimate the uncertainty of the real world because they do not include all the possible parameters we could include to model our data - the inclusion of more parameters is expected to make the posterior more precise.
- Posterior distributions will also underestimate the uncertainty in the real world because they provide a 'small world' picture of a 'large world' problem (this is the language of Richard McElreath to describe models). Many similar models could explain the data, meaning there is greater uncertainty than displayed by the posterior.
To explore these topics, model checking and expansion can be used. We can, for instance, use posterior predictive checks to see if what our model predicts matches our data, and if our parameter estimates (e.g. the differences between two groups) are consistent with our prior knowledge (if available). We can also change prior distributions, or use more robust models (such as variable selection, as outlined clearly in Kruschke (2015) 'Doing Bayesian Data Analysis'). If these models result in large differences to the posterior compared to our other model, we may change how strongly we believe in our posterior.
In sum, Bayesian analysis cannot tell us the true model any more than frequentist methods. And the role of model checking is applicable to all other areas of statistics (for instance, there are equivalent structural equation models and latent class models). In Bayesian analysis, we must check our model against our prior knowledge and also assess the sensitivity of our model to other prior distributions. If we are satisfied with our model afterwards, we can conclude that our model is not as bad as other possible models.
EDIT What can we do if our model is 'wrong'? NB. All models are wrong, just some are less bad than others. However, currently, from my reading, there are two ways that jump out.
We can choose to use information criteria, such as the Deviance Information Criteria (similar to the AIC, BIC etc.) or WAIC, to help us choose the most parsimonious model, which penalises for the number of parameters. However, there are ontological debates about what the 'best model' even means.
I personally like the 'parameter estimation' approach advocated by John Kruschke. From the start, this assumes that the model we include is the model we are interested in, and that our goal is not just black and white conclusions, but gaining precise estimates about the (small) world. Then, we can look at how precise our parameter estimates are, including if they include null values or comparison values of interest. To protect against false alarms, we can include a region of practical equivalence or ROPE around null or comparison values, which are practically equivalent to the null/comparison value. For instance, we can include a ROPE of 0.05 around 0, and take the 95% HDI of each parameter. Kruschke has shown the utility of this approach in his book (see above), but also:
Kruschke et al. (2011) 'Bayesian assessment of null values via parameter estimation and model comparison'
And in this YouTube series: https://www.youtube.com/watch?v=lh5btlAvrLs