Bayesian inference with an arbitrary prior

A classical problem in Bayesian inference arises when we wish to learn about (say) the fraction $$\theta$$ of balls in an urn that are white; and do so by sampling from the urn with replacement. In such a situation, the likelihood of the data is binomial: i.e. for any $$\theta$$, the chance that $$y$$ balls out of the $$n$$ sampled are black is proportional to $$\theta^y (1 - \theta)^{y-n}$$. In the textbook I have been reading, the prior is assumed to take the form of a beta distribution. I am wondering, however, about what we can say without making any particular assumptions about the shape of the individual's prior.

For example, it seems to me that the following should be true:

1. The posterior expectation of $$\theta$$ is increasing in the fraction of balls sampled that are black $$y/n$$. Equivalently, holding $$n$$ fixed, the posterior expectation is higher the greater is $$y$$.
2. As $$n \rightarrow \infty$$, the posterior expectation tends to the true proportion $$\theta$$.

Is there a textbook (or any other resource) that provides a discussion of topics like these (without making any assumptions about the shape of the prior)? Alternately, is there an easy way to prove that (1) and (2) must be true?

• Prior must have support including [0,1]. Most logically, support of prior is precisely [0,1]. Maybe start with a beta and branch from there. Given sensible prior: as n→∞ likelihood from data will determine posterior. – Jul 7 '20 at 18:45
• I think this is covered in chapter 4 of gelman's book that you refer to Aug 17 '20 at 16:08

This topic is probably most known as Bayesian consistency or Bayesian asymptotics, an area that, broadly speaking, studies the following class of problems:

Suppose that there is a "true" data generating density (for instance data are generated i.i.d from a parametric distribution) and we assume a prior over this density (for instance by assuming a prior over the parameters of the data generating density), can we show that with enough samples the posterior will converge to the true data generating density?

The basic result in this field is called Doob's theorem, which basically says that this will happen almost for all parameters in the support of the prior, so that if one assumes a parametric model and the prior is not "too stupid", than this convergence will occur. For a detailed reference on Doob's theorem see https://arxiv.org/abs/1801.03122

On the other hand, if the model is nonparametrico, stuff becomes more and more difficult, mainly because the prior null-set (I.e. with prior mass equal zero) can actually be huge. A nice (but complex book) that summarizes the last 30 years of research in this field is "Fundamentals of Bayesian Nonparametric Inference" by Ghosal and van see Vaarr

• Thanks for the reference! I don't suppose you have a reference for (1) (which is not a question about asymptotics)? Jul 9 '20 at 10:45
• About (1), this is highly dependent on the prior. Suppose that the prior is a delta in 0.1, than the posterior will be as well Jul 9 '20 at 11:00
• I think what @mariob6 is saying that if you have a "stubborn" (dogmatic) prior that $\theta$ is, say, 1/3, then the data will not change that as you put all your prior probability there and hence 0 elsewhere, and $0\times likelihood=0$. Jul 9 '20 at 11:55
• For (1), it seems you'd need to do something like $\partial E(\theta|y)/\partial y=\partial \int y p(\theta)f(y|\theta)/(\int p(\theta)f(y|\theta)d\theta)d\theta/\partial y$. It seems OK to interchange derivative and integral, but then I currently do not see how to establish results indepently of $p(\theta)$, as the prior also influences the form of the marginal likelihood in the denominator. Jul 9 '20 at 12:10
• Also note that y is discrete, so that deriving w.r.t. is not well defined Jul 9 '20 at 17:36

Here is an answer to item 1 that shows the result about the posterior mean increasing in the data.

It is derived for exponential families (one parameter, canonical parametrization) based on Tweedie's formula, as cited in Efron (JASA 2011):

Consider a prior $$g$$ and an exponential family $$f_\eta(z)$$, $$\eta ∼ g(\cdot), \quad z|\eta ∼ f_\eta(z) = e^{\eta z−\psi(\eta)} f_0(z)$$ Here, $$\eta$$ is the natural or canonical parameter of the family, $$\psi(\eta)$$ the cumulant generating function or cgf (which makes $$f_\eta(z)$$ integrate to 1), and $$f_0(z)$$ the density when $$\eta = 0$$.

Bayes rule provides the posterior density of $$\eta$$ given $$z$$, $$g(\eta|z) = f_\eta(z)g(\eta)/f(z)$$ where $$f(z)$$ is the marginal density $$f(z) = \int f_\eta(z)g(\eta) d\eta$$ Then the first display gives $$g(\eta|z) = e^{z\eta−\lambda(z)} g(\eta)e^{-\psi(\eta)}$$ where $$\lambda(z) = \log (f(z)/f_0(z))$$ This expression represents an exponential family with canonical parameter $$z$$ and cgf $$\lambda(z)$$. Differentiating $$\lambda(z)$$ hence yields the posterior cumulants of $$\eta$$ given $$z$$, see e.g. here. In particular, $$E(\eta|z) = \lambda'(z),\qquad var(\eta|z) = \lambda''(z)$$ Hence, $$E(\eta|z)$$ is an increasing function of $$z$$, as $$\frac{\partial E(\eta|z)}{\partial z}=\lambda''(z)=var(\eta|z)\geq0.$$