Posterior of one observation transform into posterior of several observations

Question

Suppose $\mu$ has prior distribution $\mathcal{N}(M, A)$ and $x |\mu \sim \mathcal{N}(\mu, 1)$

After one observation, the posterior is $$\mu|x \sim \mathcal{N}(M + B(x-M), B), \tag{1}$$ where $B \overset{\text{(def)}}{=} \dfrac{A}{A+1}$.

After $n$ independent observations $\mathbf{x} \overset{\text{(def)}}{=}x_1, \dotsc, x_n | \mu\overset{\perp \!\!\perp}{\sim} \mathcal{N}(\mu, 1)$, the posterior is $$\mu|\mathbf{x} \sim \mathcal{N}\left(M + B_n(\bar{x} - M), \frac{1}{n}B_n \right), \tag{2}$$ where $B_n \overset{\text{(def)}}{=} \dfrac{nA}{nA + 1}$.

Is there the general method to calculate the posterior after several observations, given the prior and the posterior after one observation? Is there a path that can take me from <(1) , prior> to (2) in different situations?

It should be just manipulating Bayes' rule...someone help me out here. Thanks

Answer 1

Ah! Beautiful question. First, let's look at the mean value of the posterior distribution for one sample vs $n$ samples. \begin{align} B &= \frac{A}{A + 1} \\ B_n &= \frac{nA}{nA + 1} = \frac{A}{A + \frac{1}{n}} \end{align} where in the second equation I have divided the numerator and denominator by $n$. Note that this does not change the expression of $B_n$. Let's begin with one sample case and look at the mean value of the posterior distribution \begin{align} M + B(\bar{x} - M) = \frac{M}{A+1} + \frac{A \bar{x}}{A + 1}, \end{align} where $A$ is the variance of the prior distribution. The above expression is a weighted average of the prior mean $M$ and the true sample mean $\bar{x}$, where the weights $1/(A + 1)$ and $A/(A+1)$ determine how much I should trust my sample mean vs prior mean.

Now let us consider the case where we collect large number samples. As $n \rightarrow \infty$, $1/n \rightarrow 0$. Therefore, $B_n \rightarrow 1$. Hence, the mean value of the posterior distribution converges to the sample mean $\bar{x}$. This is given as follows: \begin{align} M + B_n(\bar{x} - M) &\approx M + (\bar{x} - M) \\ &= \bar{x} \end{align} So, as I get more samples, I can estimate the mean by taking an average of the samples $\bar{x}$ instead of relying on the prior distribution mean $M$. In some sense, I become less dependent on the prior distribution sample mean as I get more samples. In the same way, you can also look at what happens to the variance of the posterior distribution as you get more samples. I will leave that as an exercise for you to solve.

Answer 2

Exponential families are notable precisely because there is a general method like the one you’re (I’m) searching for. Unfortunately this is somewhat obscured in the Wikipedia article.

Firstly in exponential families we are dealing with families of measures that are by definition absolutely continuous with respect to one another -- they are exponential tiltings of a given measure. So we are free to talk in terms of densities${}^1$ with respect to some fixed measure (maybe Lebesgue measure, for instance).

Suppose the prior for $\mu$ has density $g(\mu)$, and $x |\mu$ has density $f_{\mu}(x)$. Then the posterior on $x_1, \dotsc, x_n$ independent observations is $\mu|\mathbf{x}$ has density $\propto g(\mu) \prod_{i=1}^nf_\mu(x_i)$

Exponential families can simplify this. If we are in an exponential family, we can find the (unnormalized) posterior after $n$ observations with a simple adjustment, using no other information but the average $\bar{y} = \frac{1}{n}(y(x_1) + \dotsb + y(x_n))$ of the sufficient statistic.

In more detail, suppose the prior for $\mu$ has density $g(\mu)$, and $x |\mu$ has density $e^{\alpha y - \psi (\alpha)}f_{\mu_0}(x)$ for $y = t(x)$ and $\alpha = \alpha(\mu)$ the natural parameter. Then the posterior $\mu | \mathbf{x}$ has density $\propto g(\mu)e^{\alpha \left( \sum y_i \right) - n \cdot \psi(\alpha)} \prod f_{\mu_0}(x_i)$, and we can remove the $\prod f_{\mu_0}(x_i)$ since we are satisfying ourself with an un-normalized density, and $\prod f_{\mu_0}(x_i)$ does not depend on $\mu$. (The quantity $\mu_0$ is some fixed constant.)

$$\mu | \mathbf{x} \propto g(\mu)e^{\alpha \left( \sum y_i \right) - n \cdot \psi(\alpha)} \,\,\text{ AKA }\,\,g(\mu)e^{n \cdot (\alpha \bar{y} - \psi(\alpha))}$$

See here for another form.

Relevant is this from page 69 of Efron/Hastie's Computer Age Statistical Inference

In 1935–36, a trio of authors, working independently in different countries, Pitman, Darmois, and Koopmans, showed that exponential families are the only ones that enjoy fixed-dimensional sufficient statistics under repeated independent sampling.

---

1: See I-projection, where an exponentially-tilted version of a measure $\mu$ is the orthogonal projection of $\mu$ in a certain inner product space, onto $\{\nu \mid \nu \ll \mu \text{ and } C\}$, where $C$ is some constraint.