Suppose $\mu$ has prior distribution $\mathcal{N}(M, A)$ and $x |\mu \sim \mathcal{N}(\mu, 1)$

After one observation, the posterior is $$\mu|x \sim \mathcal{N}(M + B(x-M), B), \tag{1}$$ where $B \overset{\text{(def)}}{=} \dfrac{A}{A+1}$.

After $n$ independent observations $\mathbf{x} \overset{\text{(def)}}{=}x_1, \dotsc, x_n | \mu\overset{\perp \!\!\perp}{\sim} \mathcal{N}(\mu, 1)$, the posterior is $$\mu|\mathbf{x} \sim \mathcal{N}\left(M + B_n(\bar{x} - M), \frac{1}{n}B_n \right), \tag{2}$$ where $B_n \overset{\text{(def)}}{=} \dfrac{nA}{nA + 1}$.

Is there the general method to calculate the posterior after several observations, given the prior and the posterior after one observation? Is there a path that can take me from <(1) , prior> to (2) in different situations?

It should be just manipulating Bayes' rule...someone help me out here. Thanks

  • 2
    $\begingroup$ You need to look at how the likelihood for n observations relates to the likelihood for one. Once you know how the likelihood 'updates', you can (when you have conjugacy at least) see how to update the posterior. $\endgroup$
    – Glen_b
    Aug 4 at 4:31
  • 1
    $\begingroup$ The posterior for $n$ observations is acting like the prior for the $n+1$th observation. Beyond exponential families, there is no other simple updating formula. $\endgroup$
    – Xi'an
    Aug 4 at 5:54
  • 1
    $\begingroup$ @Xi'an Thank you, that was the crucial piece of info I was trying to remember. This possibility is what characterizes exponential families. $\endgroup$
    – Eric Auld
    Aug 4 at 7:01

Ah! Beautiful question. First, let's look at the mean value of the posterior distribution for one sample vs $n$ samples. \begin{align} B &= \frac{A}{A + 1} \\ B_n &= \frac{nA}{nA + 1} = \frac{A}{A + \frac{1}{n}} \end{align} where in the second equation I have divided the numerator and denominator by $n$. Note that this does not change the expression of $B_n$. Let's begin with one sample case and look at the mean value of the posterior distribution \begin{align} M + B(\bar{x} - M) = \frac{M}{A+1} + \frac{A \bar{x}}{A + 1}, \end{align} where $A$ is the variance of the prior distribution. The above expression is a weighted average of the prior mean $M$ and the true sample mean $\bar{x}$, where the weights $1/(A + 1)$ and $A/(A+1)$ determine how much I should trust my sample mean vs prior mean.

Now let us consider the case where we collect large number samples. As $n \rightarrow \infty$, $1/n \rightarrow 0$. Therefore, $B_n \rightarrow 1$. Hence, the mean value of the posterior distribution converges to the sample mean $\bar{x}$. This is given as follows: \begin{align} M + B_n(\bar{x} - M) &\approx M + (\bar{x} - M) \\ &= \bar{x} \end{align} So, as I get more samples, I can estimate the mean by taking an average of the samples $\bar{x}$ instead of relying on the prior distribution mean $M$. In some sense, I become less dependent on the prior distribution sample mean as I get more samples. In the same way, you can also look at what happens to the variance of the posterior distribution as you get more samples. I will leave that as an exercise for you to solve.

  • $\begingroup$ Thank you! That provides some good intuition. What I was looking for is a general method of which this is one instance: given a prior and a one-sample posterior, how do you get to an n-sample posterior? $\endgroup$
    – Eric Auld
    Aug 4 at 4:03
  • 1
    $\begingroup$ Well, the expressions would depend on assumptions you make. For example, if the samples are drawn from Gaussian and the prior is Gaussian, then you'd get a Gaussian posterior distribution (as stated in the above example). For some choices of distributions, you might not get a closed-form expression. So, it all depends on how you model the data. $\endgroup$
    – Maxtron
    Aug 4 at 5:14

Exponential families are notable precisely because there is a general method like the one you’re (I’m) searching for. Unfortunately this is somewhat obscured in the Wikipedia article.

Firstly in exponential families we are dealing with families of measures that are by definition absolutely continuous with respect to one another -- they are exponential tiltings of a given measure. So we are free to talk in terms of densities${}^1$ with respect to some fixed measure (maybe Lebesgue measure, for instance).

Suppose the prior for $\mu$ has density $g(\mu)$, and $x |\mu$ has density $f_{\mu}(x)$. Then the posterior on $x_1, \dotsc, x_n$ independent observations is $\mu|\mathbf{x}$ has density $\propto g(\mu) \prod_{i=1}^nf_\mu(x_i)$

Exponential families can simplify this. If we are in an exponential family, we can find the (unnormalized) posterior after $n$ observations with a simple adjustment, using no other information but the average $\bar{y} = \frac{1}{n}(y(x_1) + \dotsb + y(x_n))$ of the sufficient statistic.

In more detail, suppose the prior for $\mu$ has density $g(\mu)$, and $x |\mu$ has density $e^{\alpha y - \psi (\alpha)}f_{\mu_0}(x)$ for $y = t(x)$ and $\alpha = \alpha(\mu)$ the natural parameter. Then the posterior $\mu | \mathbf{x}$ has density $\propto g(\mu)e^{\alpha \left( \sum y_i \right) - n \cdot \psi(\alpha)} \prod f_{\mu_0}(x_i)$, and we can remove the $\prod f_{\mu_0}(x_i)$ since we are satisfying ourself with an un-normalized density, and $\prod f_{\mu_0}(x_i)$ does not depend on $\mu$. (The quantity $\mu_0$ is some fixed constant.)

$$\mu | \mathbf{x} \propto g(\mu)e^{\alpha \left( \sum y_i \right) - n \cdot \psi(\alpha)} \,\,\text{ AKA }\,\,g(\mu)e^{n \cdot (\alpha \bar{y} - \psi(\alpha))}$$

See here for another form.

Relevant is this from page 69 of Efron/Hastie's Computer Age Statistical Inference

In 1935–36, a trio of authors, working independently in different countries, Pitman, Darmois, and Koopmans, showed that exponential families are the only ones that enjoy fixed-dimensional sufficient statistics under repeated independent sampling.

1: See I-projection, where an exponentially-tilted version of a measure $\mu$ is the orthogonal projection of $\mu$ in a certain inner product space, onto $\{\nu \mid \nu \ll \mu \text{ and } C\}$, where $C$ is some constraint.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.