Regarding the book The Bayesian Choice I understand most of chapter three on exponential families, but there are two parts I have trouble understanding.

The first is

Consider$$f(x|\theta)=h(x)\exp(\theta \cdot x - \psi(\theta))$$a generic distribution from an exponential family, then a proposition is that a conjugate family for $\theta$ is given by $$\pi(\theta|\mu , \lambda)=K(\mu,\lambda)\exp(\theta \cdot \mu - \lambda \psi(\theta))$$ where $K$ represents the normalising constant and the corresponding posterior is $\pi(\theta|\mu+x,\lambda+1)$ (under certain conditions).

I don't really understand this or how it is applied. For example consider a $Poisson(\lambda)$ density $$f(x|\lambda)=\frac{1}{x!}\exp(x\ln\lambda-\lambda)$$ Our sufficient statistic is $x$, and we can write in the form above either as $\psi(\lambda)=\lambda$ or using natural parameter form $\psi(n)=\exp(n)$ as our natural parameter is $n=\log \lambda$

But I don't understand the proposition above, how it is applied or used.

Second, I don't understand the following related proposition from the same page

If $\Theta$ is an open set in $\mathbb{R^{k}}$ and $\theta$ has prior $$\pi_{\lambda,x_{0}}(\theta)\propto\exp(\theta \cdot x_{0}-\lambda \psi(\theta))$$ with $x_{0}$ in $X$ then $$\text{E}[\nabla \psi (\theta)]=\frac{x_{0}}{\lambda}$$ and for $n$ iid observations the conditional posterior expectation over all is $\frac{x_{0}+n \bar x}{\lambda+n}$

Looking for any help understanding these or examples of how they are used/why they are important etc.


1 Answer 1


Quoting verbatim from my book

It is always possible to reduce an exponential family to a standard and minimal form of dimension $m$, and this dimension $m$ does not depend on the chosen parameterisation (Brown, 1986, pp. 13-16). (See Exercise 3.20 for an example of a non-regular exponential family.)

Natural exponential families can also be rewritten under the form \begin{equation} f(x|\theta) = h(x) e^{\theta.x -\psi(\theta)} \tag{3.7} \end{equation} and $\psi(\theta)$ is called the cumulant generating function.

This means that, whatever the original parameterisation of the exponential family, e.g., a Poisson $\mathcal{P}(\lambda)$, there exists a reparameterisation in $\theta$ that looks like the above. \begin{align*} f(x|\lambda) &= \frac{\lambda^x}{x!}e^{-\lambda}\\ &= \frac{1}{x!}\exp\{\underbrace{\log(\lambda)}_\text{this is $\theta$}\cdot x-\underbrace{\lambda}_\text{this is $\exp(\theta)$}\}\}\\ &= \frac{1}{x!}\exp\{\theta\cdot x-\underbrace{\exp(\theta)}_\text{this is $\psi(\theta)$}\}\\ &= \underbrace{f^*(x|\theta)}_{\substack{\text{same density}\\ \text{new parameterisation}}} \end{align*} This means that the conjugate prior on the natural parameter $\theta\in\mathbb{R}$ is of the form $$\pi(\theta|x_0,\eta)\propto\exp\{\theta\cdot x_0-\eta\exp(\theta)\}$$ and hence after a change of variable back to $\lambda=\exp(\theta)$, we have that a conjugate prior on the standard parameter $\lambda$ is $$\pi(\lambda|x_0,\eta)\propto \lambda^{-1} \exp\{\log(\lambda)\cdot x_0-\eta\lambda\}=\lambda^{x_0-1}\,e^{-\eta\lambda}$$ which produces the Gamma $\mathcal{G}a(x_0,\eta)$ distribution as a conjugate prior.

Similarly, for a Binomial $\mathcal{B}in(n,p)$ distribution, \begin{align*} f(x|p) &= {n \choose x}\,p^x(1-p)^{n-x}\\ &= {n \choose x}\exp\{\underbrace{\log\{p/(1-p)\}}_\text{this is $\theta$}\cdot x+\underbrace{n\log(1-p)}_\text{this is $-n\log\{1+\exp(\theta)\}$}\}\}\\ &= {n \choose x}\exp\{\theta\cdot x-\underbrace{n\log\{1+\exp(\theta)\}}_\text{this is $\psi(\theta)$}\}\\ &= \underbrace{f^*(x|\theta)}_{\substack{\text{same density}\\ \text{new parameterisation}}} \end{align*} Meaning that a conjugate prior on the natural parameter $\theta$ of the Binomial distribution is of the form $$\pi(\theta|x_0,\lambda) \propto \exp\{\theta\cdot x_0-\underbrace{\lambda\log\{1+\exp(\theta)\}}_\text{$n$ incorporated in $\lambda$}\}$$ For the standard parameter $p\in(0,1)$, this is yet another change of variable from $\theta=\log(p/\{1-p\})$ to $p$: $$\pi(p|x_0,\lambda) \propto \underbrace{\frac{1}{p(1-p)}}_\text{Jacobian}\,\left(\frac{p}{1-p}\right)^{x_0}\,\exp\{\lambda\log(1-p)\}=p^{x_0-1}\,(1-p)^{\lambda-x_0-1}$$ which returns a Beta $\mathcal{B}e(x_0,\lambda-x_0)$ conjugate distribution (with the constraints $x_0>0$ and $\lambda>x_0$).

Lemma 3.2 If $\theta \in\, \stackrel{\circ }{N}$, the interior set of $N$, the cumulant generating function $\psi$ is $\mathcal{C}^{\infty}$ and $$ \mathbb{E}_\theta[x] = \nabla \psi(\theta), \qquad \mathrm{cov} (x_i,x_j) = {\partial^2 \psi\over \partial \theta_i \partial \theta_j} (\theta), $$ where $\nabla$ denotes the gradient operator.

This is only true for the natural parameterisation (3.7). Which is also used below:

(...) Consider $f(x|\theta) = h(x) e^{\theta\cdot x - \psi(\theta)}$, a generic distribution from an exponential family. It then allows for a conjugate family, as shown by the following result (whose proof is straightforward).

Proposition 3.3 A conjugate family for $f(x|\theta)$ is given by $$ \pi(\theta|\mu,\lambda)=K(\mu,\lambda)\,e^{\theta\cdot\mu-\lambda\psi(\theta)}, \tag{3.8} $$ where $K(\mu,\lambda)$ is the normalizing constant of the density. The corresponding posterior distribution is $\pi(\theta|\mu+x,\lambda+1)$.

This means that, when the prior is the conjugate distribution indexed by the hyper-parameters$$(\mu,\lambda)$$the posterior with one observation is the conjugate distribution indexed by the hyper-parameters$$(\mu+x,\lambda+1)$$and for $n$ observations the conjugate distribution indexed by the hyper-parameters$$(\mu+\sum x_i,\lambda+n)$$This explains for the last part:

Proposition 3.4 If $\Theta$ is an open set in $\mathbb{R}^k$ and $\theta$ has the prior distribution $$ \pi_{\lambda,x_0}(\theta) \propto e^{\theta\cdot x_0 -\lambda \psi(\theta)} $$ with $x_0 \in \mathcal{X}$, then $$ \mathbb{E}^\pi[\overbrace{\xi(\theta)}^{\substack{\text{notation}\\ \text{for $\mathbb{E}_\theta[X]$}}}]=\mathbb{E}^\pi[\overbrace{\nabla \psi(\theta)}^{\substack{\text{as shown in}\\ \text{Lemma 3.2}}}]={x_0\over\lambda}. $$ Therefore, if $x_1,\ldots,x_n$ are i.i.d. $f(x|\theta)$, \begin{equation} \mathbb{E}^\pi[\xi(\theta)|x_1,\ldots,x_n] = \underbrace{{x_0 + n {\bar x} \over \lambda+n}.}_\text{by conjugacy} \tag{3.9} \end{equation}

The proof is done by an integration by part \begin{align*}\int \nabla \psi(\theta)\,\exp\{-\psi(\theta)+\theta\cdot x_0\}\,\text{d}\theta&=-\int \nabla \exp\{-\psi(\theta)\} \,\exp\{\theta\cdot x_0\}\,\text{d}\theta\\ &=\int \exp\{-\psi(\theta)\}\,\nabla\exp\{\theta\cdot x_0\}\,\text{d}\theta\\ &= \int x_0 \,\exp\{-\psi(\theta)+\theta\cdot x_0\}\,\text{d}\theta \end{align*} Applying this Proposition 3.4 to the Poisson case means (i) getting back to the natural parameterisation as $\theta=\log\{\lambda\}$; (ii) taking the derivative of $\psi(\theta)=\exp\{\theta\}$, i.e. $\psi'(\theta)=\exp\{\theta\}=\lambda=\xi(\theta)=\mathbb{E}_\theta[X]$; (iii) deducing that the posterior expectation of the mean of $X$ is $$\mathbb{E}^\pi[\lambda|x,x_0,\eta]=\mathbb{E}^\pi[\lambda|x,x_0,\eta]=\frac{x+x_0}{\lambda+1}$$ and $$\mathbb{E}^\pi[\lambda|x_1,\ldots,x_n,x_0,\eta]=\frac{n\bar{x}_n+x_0}{\lambda+n}$$ Similarly, for the Binomial case, (i) use the natural parameterisation $\theta=\log\{p/(1-p)\}$; (ii) compute the derivative of $\psi(\theta)=\log\{1+e^\theta\}$, i.e., $$\psi'(\theta)=e^\theta/\{1+e^\theta\}=p=\mathbb{E}_p[X]$$ (iii) deduce that the posterior expectation of $p$ under a conjugate prior is $$\mathbb{E}^\pi[\psi'(\theta)|x,x_0,\lambda]=\mathbb{E}^\pi[p|x,x_0,\lambda]=\underbrace{\frac{x_0+x}{\lambda+n}}_{{\text{remember $n$}\\ \text{integrated in $\lambda$}}}$$

[Surprisingly, this is the second week in a row that I find questions on X validated connected with what I just taught in class!]

  • $\begingroup$ Thank you. So how would I interpret (3.8) for example in the $Bin(n,\theta)$ , $Beta(\alpha,\beta)$, where we go from $(\alpha,\beta)$ to $(\alpha+x,\beta+n-x)$ $\endgroup$
    – Quality
    Dec 13, 2018 at 19:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.