Bayesian inference with an arbitrary prior

Question

A classical problem in Bayesian inference arises when we wish to learn about (say) the fraction $\theta$ of balls in an urn that are white; and do so by sampling from the urn with replacement. In such a situation, the likelihood of the data is binomial: i.e. for any $\theta$, the chance that $y$ balls out of the $n$ sampled are black is proportional to $\theta^y (1 - \theta)^{y-n}$. In the textbook I have been reading, the prior is assumed to take the form of a beta distribution. I am wondering, however, about what we can say without making any particular assumptions about the shape of the individual's prior.

For example, it seems to me that the following should be true:

1. The posterior expectation of $\theta$ is increasing in the fraction of balls sampled that are black $y/n$. Equivalently, holding $n$ fixed, the posterior expectation is higher the greater is $y$.
2. As $n \rightarrow \infty$, the posterior expectation tends to the true proportion $\theta$.

Is there a textbook (or any other resource) that provides a discussion of topics like these (without making any assumptions about the shape of the prior)? Alternately, is there an easy way to prove that (1) and (2) must be true?

Many thanks in advance!

Answer 1

This topic is probably most known as Bayesian consistency or Bayesian asymptotics, an area that, broadly speaking, studies the following class of problems:

Suppose that there is a "true" data generating density (for instance data are generated i.i.d from a parametric distribution) and we assume a prior over this density (for instance by assuming a prior over the parameters of the data generating density), can we show that with enough samples the posterior will converge to the true data generating density?

The basic result in this field is called Doob's theorem, which basically says that this will happen almost for all parameters in the support of the prior, so that if one assumes a parametric model and the prior is not "too stupid", than this convergence will occur. For a detailed reference on Doob's theorem see https://arxiv.org/abs/1801.03122

On the other hand, if the model is nonparametrico, stuff becomes more and more difficult, mainly because the prior null-set (I.e. with prior mass equal zero) can actually be huge. A nice (but complex book) that summarizes the last 30 years of research in this field is "Fundamentals of Bayesian Nonparametric Inference" by Ghosal and van see Vaarr

Answer 2

Here is an answer to item 1 that shows the result about the posterior mean increasing in the data.

It is derived for exponential families (one parameter, canonical parametrization) based on Tweedie's formula, as cited in Efron (JASA 2011):

Consider a prior $g$ and an exponential family $f_\eta(z)$, $$\eta ∼ g(\cdot), \quad z|\eta ∼ f_\eta(z) = e^{\eta z−\psi(\eta)} f_0(z)$$ Here, $\eta$ is the natural or canonical parameter of the family, $\psi(\eta)$ the cumulant generating function or cgf (which makes $f_\eta(z)$ integrate to 1), and $f_0(z)$ the density when $\eta = 0$.

Bayes rule provides the posterior density of $\eta$ given $z$, $$g(\eta|z) = f_\eta(z)g(\eta)/f(z)$$ where $f(z)$ is the marginal density $$f(z) = \int f_\eta(z)g(\eta) d\eta$$ Then the first display gives $$g(\eta|z) = e^{z\eta−\lambda(z)} g(\eta)e^{-\psi(\eta)}$$ where $$\lambda(z) = \log (f(z)/f_0(z))$$ This expression represents an exponential family with canonical parameter $z$ and cgf $\lambda(z)$. Differentiating $\lambda(z)$ hence yields the posterior cumulants of $\eta$ given $z$, see e.g. here. In particular, $$E(\eta|z) = \lambda'(z),\qquad var(\eta|z) = \lambda''(z)$$ Hence, $E(\eta|z)$ is an increasing function of $z$, as $$\frac{\partial E(\eta|z)}{\partial z}=\lambda''(z)=var(\eta|z)\geq0.$$