1
$\begingroup$

I am puzzled about model improvement in implementing Bayesian statistics/inference.

Normally, we will use a fixed model in bayesian statistics, e.g. normal distribution with parameter mean and sd. with more extra or new data, we will update the mean and sd but we keep using normal distribution.

If the Bayesian inference from posterior distribution is not consistent with observed data because wrong hypothesis model is chosen, how to solve it? There are infinitely many possible models, how to find the most suitable model given data?

More Information

From Bayesian Data Analysis (Andrew Gelman), it states that if the actual underlying model is not normal distribution but we assume a wrong model, using bayesian statistics will not lead us to the any useful data analysis result. The book mentions about model expansion and model improvement (other than updating pre-assumed model's parameters) which can sovle the above situation. (Bayesian Data Analysis 3rd Edition Chapter 7.5-7.6)

But I do not understand what the book means. Can anyone explain or elaborate more about the model expansion and model improvement mentioned in the book?

$\endgroup$

closed as unclear what you're asking by Tim, Richard Hardy, Greenparker, gung, John Jul 11 '16 at 16:49

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ I do not understand your question... What improvement are you writing about? $\endgroup$ – Tim Jul 11 '16 at 7:34
  • $\begingroup$ normally, we will use a fixed model in bayesian statistics, say, normal distribution with parameter mean and sd. with more extra or new data, we update the mean and sd but we keep using normal distribution. From BDA, it states that if the actual underlying model is not normal distribution but we assume a wrong model, using bayesian statistics will not lead us to the any useful data analysis result. The book mentions about model expansion and model improvement but I do not really understand what the book means. I hope to know how to solve the above situation. $\endgroup$ – BenBTC Jul 11 '16 at 7:38
  • $\begingroup$ Can you edit and provide the quote that you do not understand and what is unclear? Without it I am afraid that you could get only a non-answer like "by model improvement we mean improving our model"... $\endgroup$ – Tim Jul 11 '16 at 7:45
  • $\begingroup$ Thank Tim! I will change my question. Do you think anything to add will be better? $\endgroup$ – BenBTC Jul 11 '16 at 7:51
  • 1
    $\begingroup$ Thanks but still, as stated like this this question can be answered only by a person who (1) has this book at hand, (2) is willing to read the chapter and look there for some part that you are referring to. Moreover, such answer would be useless for anyone else but you since people not having this book won't know what problem is the answer referring to. Because of this I'm voting to close this question as unclear. Please check: stats.stackexchange.com/help/how-to-ask $\endgroup$ – Tim Jul 11 '16 at 9:35
2
$\begingroup$

I think this quote perhaps highlights the main issue (Gelman et al. 2014; chapter 7, section 7.5. pg. 184):

In general, the posterior distribution of the model parameters can either overestimate or underestimate different aspects of ‘true’ posterior uncertainty. The posterior distribution typically overestimates uncertainty in the sense that one does not, in general, include all of one’s substantive knowledge in the model; hence the utility of checking the model against one’s substantive knowledge. On the other hand, the posterior distribution underestimates uncertainty in two senses: first, the assumed model is almost certainly wrong—hence the need for posterior model checking against the observed data—and second, other reasonable models could have fit the observed data equally well, hence the need for sensitivity analysis.

The key message is that 'every model is wrong', i.e. our statistical models are just abstractions of the real world, and we should not assume that they are exact representations of the world. For instance, Richard McElreath in his book Statistical Rethinking (an excellent book on Bayesian analysis, and more accessible that Gelman et al. IMO) refers to statistical models as 'statistical golems' that spit out results of our analyses.

Therefore, given a posterior distribution, Gelman et al. (2014) are discussing ways of checking just how wrong are models are.

  • Posterior distributions will likely overestimate the uncertainty of the real world because they do not include all the possible parameters we could include to model our data - the inclusion of more parameters is expected to make the posterior more precise.
  • Posterior distributions will also underestimate the uncertainty in the real world because they provide a 'small world' picture of a 'large world' problem (this is the language of Richard McElreath to describe models). Many similar models could explain the data, meaning there is greater uncertainty than displayed by the posterior.

To explore these topics, model checking and expansion can be used. We can, for instance, use posterior predictive checks to see if what our model predicts matches our data, and if our parameter estimates (e.g. the differences between two groups) are consistent with our prior knowledge (if available). We can also change prior distributions, or use more robust models (such as variable selection, as outlined clearly in Kruschke (2015) 'Doing Bayesian Data Analysis'). If these models result in large differences to the posterior compared to our other model, we may change how strongly we believe in our posterior.

In sum, Bayesian analysis cannot tell us the true model any more than frequentist methods. And the role of model checking is applicable to all other areas of statistics (for instance, there are equivalent structural equation models and latent class models). In Bayesian analysis, we must check our model against our prior knowledge and also assess the sensitivity of our model to other prior distributions. If we are satisfied with our model afterwards, we can conclude that our model is not as bad as other possible models.

EDIT What can we do if our model is 'wrong'? NB. All models are wrong, just some are less bad than others. However, currently, from my reading, there are two ways that jump out.

  • We can choose to use information criteria, such as the Deviance Information Criteria (similar to the AIC, BIC etc.) or WAIC, to help us choose the most parsimonious model, which penalises for the number of parameters. However, there are ontological debates about what the 'best model' even means.

  • I personally like the 'parameter estimation' approach advocated by John Kruschke. From the start, this assumes that the model we include is the model we are interested in, and that our goal is not just black and white conclusions, but gaining precise estimates about the (small) world. Then, we can look at how precise our parameter estimates are, including if they include null values or comparison values of interest. To protect against false alarms, we can include a region of practical equivalence or ROPE around null or comparison values, which are practically equivalent to the null/comparison value. For instance, we can include a ROPE of 0.05 around 0, and take the 95% HDI of each parameter. Kruschke has shown the utility of this approach in his book (see above), but also:

Kruschke et al. (2011) 'Bayesian assessment of null values via parameter estimation and model comparison'

And in this YouTube series: https://www.youtube.com/watch?v=lh5btlAvrLs

$\endgroup$
  • $\begingroup$ Thanks for your answer. I am still puzzled maybe I ask not clearly. Your suggested methods will help reduce the number of parameters or seek more accurate parameters' values. Your method assumes a model is fixed first. doesn't it? E.g. normal distribution and log-normal distribution, both have 2 parameters. IC and parameter estimation will help us find the optimal parameters but cannot suggest which better model we should use. Does Bayesian statistics/inference can help us optimize model's parameters only? i.e., it cannot suggest the best model given data? How to build model given data? $\endgroup$ – BenBTC Jul 12 '16 at 5:42
  • $\begingroup$ I do not think there is a way to build the 'best' model indefinitely, since it is not known what the best model is. You have to define what your best model is first, i.e. most variance explained etc. $\endgroup$ – user3237820 Jul 15 '16 at 10:21
  • $\begingroup$ Instead of using "best" word, how to build "appropriate" model. If we need to assume the mode first, I think it is not to build the model but fit the model. Given most, if all, data, you can the corresponding fitted value of parameters for the pre-assumed model. What is the meaning of doing data analysis? $\endgroup$ – BenBTC Jul 18 '16 at 2:04

Not the answer you're looking for? Browse other questions tagged or ask your own question.