3
$\begingroup$

In Bayesian statistics, usually we have some prior distribution $P(M)$, some observations $X$, and we compute a posterior $P(M|X)$. The observation allows us to gain information about the generating distribution, and learn about $M$.

My question is this: what if we gained less than the maximum amount of information about $M$ for some reason? This could happen if the update was somehow incomplete, like if the Bayesian update was implemented in some less-than-perfect system, or perhaps a system where there was an update cost. In that case, we would end up with a pseudo-posterior that was somewhere between $P(M)$ and $P(M|X)$. I have a few questions:

  1. Is this a thing that people have written about? Does it have a name?
  2. If so, is there a way to describe the degree of 'incompleteness' of this update?
  3. Is it equivalent to having a noisy observation of $X$? I suspect not.
|cite|improve this question
$\endgroup$
  • $\begingroup$ I don’t know where exactly you are heading with this question but I would say that information is not processed efficiently if there is a model misspecification. For example likelihood is different than the underlying truth. There are also some results regarding information bounds and saturating those bounds. $\endgroup$ – Cagdas Ozgenc Jul 18 '18 at 20:35
  • $\begingroup$ @CagdasOzgenc I am specifically thinking of cases in which Bayesian model updating is implemented in a system of some sort (e.g. the brain), where there may be physical reasons to be less-than-optimal in your updating (e.g. the cost of changing the neural representation of the model). $\endgroup$ – tom Jul 18 '18 at 21:59
  • $\begingroup$ You might be interested in the answer to this similar question. $\endgroup$ – Reinstate Monica Jul 20 '18 at 7:23
2
$\begingroup$

In a 2014-2017 paper published in Bayesian Analysis, Peter Grünwald and Thijs van Ommen show that Bayesian inference can be inconsistent under misspecification in simple linear regression problems and propose this powering-down by a Learning rate $\eta$ to recover consistency, that is, to achieve convergence to the model within the assumed family that is closest (in the Kullback-Leibler metric) to the true model. Here, misspecification is understood as heteroskedasticity, that is, dependence of the variance on the regressors. There are more precise results in the literature, as for instance the rate at which the pseudo-posterior concentrates is minimax optimal, i.e. no algorithm can dobetter in general. Choosing the power $\eta$ is operated by minimising the log-loss we expect to obtain, according to the $\eta$-generalized posterior, if we actually sample from this pseudo-posterior.

Here is a figure reproduced from this paper

enter image description here

that shows the improvement brought by this technique compared with regular Bayes, when the assumed model is wrong, as the sample size grows. Consistency happens.

This notion is connected with a larger literature on PAC-Bayesian techniques (Seeger (2002), Catoni (2007), Bissiri et al. (2016)).

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.