# Some questions about exponential families

Regarding the book The Bayesian Choice I understand most of chapter three on exponential families, but there are two parts I have trouble understanding.

The first is

Consider$$f(x|\theta)=h(x)\exp(\theta \cdot x - \psi(\theta))$$a generic distribution from an exponential family, then a proposition is that a conjugate family for $$\theta$$ is given by $$\pi(\theta|\mu , \lambda)=K(\mu,\lambda)\exp(\theta \cdot \mu - \lambda \psi(\theta))$$ where $$K$$ represents the normalising constant and the corresponding posterior is $$\pi(\theta|\mu+x,\lambda+1)$$ (under certain conditions).

I don't really understand this or how it is applied. For example consider a $$Poisson(\lambda)$$ density $$f(x|\lambda)=\frac{1}{x!}\exp(x\ln\lambda-\lambda)$$ Our sufficient statistic is $$x$$, and we can write in the form above either as $$\psi(\lambda)=\lambda$$ or using natural parameter form $$\psi(n)=\exp(n)$$ as our natural parameter is $$n=\log \lambda$$

But I don't understand the proposition above, how it is applied or used.

Second, I don't understand the following related proposition from the same page

If $$\Theta$$ is an open set in $$\mathbb{R^{k}}$$ and $$\theta$$ has prior $$\pi_{\lambda,x_{0}}(\theta)\propto\exp(\theta \cdot x_{0}-\lambda \psi(\theta))$$ with $$x_{0}$$ in $$X$$ then $$\text{E}[\nabla \psi (\theta)]=\frac{x_{0}}{\lambda}$$ and for $$n$$ iid observations the conditional posterior expectation over all is $$\frac{x_{0}+n \bar x}{\lambda+n}$$

Looking for any help understanding these or examples of how they are used/why they are important etc.

Quoting verbatim from my book

It is always possible to reduce an exponential family to a standard and minimal form of dimension $$m$$, and this dimension $$m$$ does not depend on the chosen parameterisation (Brown, 1986, pp. 13-16). (See Exercise 3.20 for an example of a non-regular exponential family.)

Natural exponential families can also be rewritten under the form $$\begin{equation} f(x|\theta) = h(x) e^{\theta.x -\psi(\theta)} \tag{3.7} \end{equation}$$ and $$\psi(\theta)$$ is called the cumulant generating function.

This means that, whatever the original parameterisation of the exponential family, e.g., a Poisson $$\mathcal{P}(\lambda)$$, there exists a reparameterisation in $$\theta$$ that looks like the above. \begin{align*} f(x|\lambda) &= \frac{\lambda^x}{x!}e^{-\lambda}\\ &= \frac{1}{x!}\exp\{\underbrace{\log(\lambda)}_\text{this is \theta}\cdot x-\underbrace{\lambda}_\text{this is \exp(\theta)}\}\}\\ &= \frac{1}{x!}\exp\{\theta\cdot x-\underbrace{\exp(\theta)}_\text{this is \psi(\theta)}\}\\ &= \underbrace{f^*(x|\theta)}_{\substack{\text{same density}\\ \text{new parameterisation}}} \end{align*} This means that the conjugate prior on the natural parameter $$\theta\in\mathbb{R}$$ is of the form $$\pi(\theta|x_0,\eta)\propto\exp\{\theta\cdot x_0-\eta\exp(\theta)\}$$ and hence after a change of variable back to $$\lambda=\exp(\theta)$$, we have that a conjugate prior on the standard parameter $$\lambda$$ is $$\pi(\lambda|x_0,\eta)\propto \lambda^{-1} \exp\{\log(\lambda)\cdot x_0-\eta\lambda\}=\lambda^{x_0-1}\,e^{-\eta\lambda}$$ which produces the Gamma $$\mathcal{G}a(x_0,\eta)$$ distribution as a conjugate prior.

Similarly, for a Binomial $$\mathcal{B}in(n,p)$$ distribution, \begin{align*} f(x|p) &= {n \choose x}\,p^x(1-p)^{n-x}\\ &= {n \choose x}\exp\{\underbrace{\log\{p/(1-p)\}}_\text{this is \theta}\cdot x+\underbrace{n\log(1-p)}_\text{this is -n\log\{1+\exp(\theta)\}}\}\}\\ &= {n \choose x}\exp\{\theta\cdot x-\underbrace{n\log\{1+\exp(\theta)\}}_\text{this is \psi(\theta)}\}\\ &= \underbrace{f^*(x|\theta)}_{\substack{\text{same density}\\ \text{new parameterisation}}} \end{align*} Meaning that a conjugate prior on the natural parameter $$\theta$$ of the Binomial distribution is of the form $$\pi(\theta|x_0,\lambda) \propto \exp\{\theta\cdot x_0-\underbrace{\lambda\log\{1+\exp(\theta)\}}_\text{n incorporated in \lambda}\}$$ For the standard parameter $$p\in(0,1)$$, this is yet another change of variable from $$\theta=\log(p/\{1-p\})$$ to $$p$$: $$\pi(p|x_0,\lambda) \propto \underbrace{\frac{1}{p(1-p)}}_\text{Jacobian}\,\left(\frac{p}{1-p}\right)^{x_0}\,\exp\{\lambda\log(1-p)\}=p^{x_0-1}\,(1-p)^{\lambda-x_0-1}$$ which returns a Beta $$\mathcal{B}e(x_0,\lambda-x_0)$$ conjugate distribution (with the constraints $$x_0>0$$ and $$\lambda>x_0$$).

Lemma 3.2 If $$\theta \in\, \stackrel{\circ }{N}$$, the interior set of $$N$$, the cumulant generating function $$\psi$$ is $$\mathcal{C}^{\infty}$$ and $$\mathbb{E}_\theta[x] = \nabla \psi(\theta), \qquad \mathrm{cov} (x_i,x_j) = {\partial^2 \psi\over \partial \theta_i \partial \theta_j} (\theta),$$ where $$\nabla$$ denotes the gradient operator.

This is only true for the natural parameterisation (3.7). Which is also used below:

(...) Consider $$f(x|\theta) = h(x) e^{\theta\cdot x - \psi(\theta)}$$, a generic distribution from an exponential family. It then allows for a conjugate family, as shown by the following result (whose proof is straightforward).

Proposition 3.3 A conjugate family for $$f(x|\theta)$$ is given by $$\pi(\theta|\mu,\lambda)=K(\mu,\lambda)\,e^{\theta\cdot\mu-\lambda\psi(\theta)}, \tag{3.8}$$ where $$K(\mu,\lambda)$$ is the normalizing constant of the density. The corresponding posterior distribution is $$\pi(\theta|\mu+x,\lambda+1)$$.

This means that, when the prior is the conjugate distribution indexed by the hyper-parameters$$(\mu,\lambda)$$the posterior with one observation is the conjugate distribution indexed by the hyper-parameters$$(\mu+x,\lambda+1)$$and for $$n$$ observations the conjugate distribution indexed by the hyper-parameters$$(\mu+\sum x_i,\lambda+n)$$This explains for the last part:

Proposition 3.4 If $$\Theta$$ is an open set in $$\mathbb{R}^k$$ and $$\theta$$ has the prior distribution $$\pi_{\lambda,x_0}(\theta) \propto e^{\theta\cdot x_0 -\lambda \psi(\theta)}$$ with $$x_0 \in \mathcal{X}$$, then $$\mathbb{E}^\pi[\overbrace{\xi(\theta)}^{\substack{\text{notation}\\ \text{for \mathbb{E}_\theta[X]}}}]=\mathbb{E}^\pi[\overbrace{\nabla \psi(\theta)}^{\substack{\text{as shown in}\\ \text{Lemma 3.2}}}]={x_0\over\lambda}.$$ Therefore, if $$x_1,\ldots,x_n$$ are i.i.d. $$f(x|\theta)$$, $$\begin{equation} \mathbb{E}^\pi[\xi(\theta)|x_1,\ldots,x_n] = \underbrace{{x_0 + n {\bar x} \over \lambda+n}.}_\text{by conjugacy} \tag{3.9} \end{equation}$$

The proof is done by an integration by part \begin{align*}\int \nabla \psi(\theta)\,\exp\{-\psi(\theta)+\theta\cdot x_0\}\,\text{d}\theta&=-\int \nabla \exp\{-\psi(\theta)\} \,\exp\{\theta\cdot x_0\}\,\text{d}\theta\\ &=\int \exp\{-\psi(\theta)\}\,\nabla\exp\{\theta\cdot x_0\}\,\text{d}\theta\\ &= \int x_0 \,\exp\{-\psi(\theta)+\theta\cdot x_0\}\,\text{d}\theta \end{align*} Applying this Proposition 3.4 to the Poisson case means (i) getting back to the natural parameterisation as $$\theta=\log\{\lambda\}$$; (ii) taking the derivative of $$\psi(\theta)=\exp\{\theta\}$$, i.e. $$\psi'(\theta)=\exp\{\theta\}=\lambda=\xi(\theta)=\mathbb{E}_\theta[X]$$; (iii) deducing that the posterior expectation of the mean of $$X$$ is $$\mathbb{E}^\pi[\lambda|x,x_0,\eta]=\mathbb{E}^\pi[\lambda|x,x_0,\eta]=\frac{x+x_0}{\lambda+1}$$ and $$\mathbb{E}^\pi[\lambda|x_1,\ldots,x_n,x_0,\eta]=\frac{n\bar{x}_n+x_0}{\lambda+n}$$ Similarly, for the Binomial case, (i) use the natural parameterisation $$\theta=\log\{p/(1-p)\}$$; (ii) compute the derivative of $$\psi(\theta)=\log\{1+e^\theta\}$$, i.e., $$\psi'(\theta)=e^\theta/\{1+e^\theta\}=p=\mathbb{E}_p[X]$$ (iii) deduce that the posterior expectation of $$p$$ under a conjugate prior is $$\mathbb{E}^\pi[\psi'(\theta)|x,x_0,\lambda]=\mathbb{E}^\pi[p|x,x_0,\lambda]=\underbrace{\frac{x_0+x}{\lambda+n}}_{{\text{remember n}\\ \text{integrated in \lambda}}}$$

[Surprisingly, this is the second week in a row that I find questions on X validated connected with what I just taught in class!]

• Thank you. So how would I interpret (3.8) for example in the $Bin(n,\theta)$ , $Beta(\alpha,\beta)$, where we go from $(\alpha,\beta)$ to $(\alpha+x,\beta+n-x)$ – Quality Dec 13 '18 at 19:11