# Exponential family and conjugate priors

Is a distribution that belongs to the exponential family necessarily conjugate prior?

• Feb 1, 2022 at 9:19

A bit of reverse engineering: for the likelihood function associated with an exponential family, with natural parameterisation $$\theta$$ and sufficient statistic $$x$$ $$\ell(\theta|x)\propto \exp\{\theta^\prime x-\tau(\theta)\}\quad x\in\mathcal X,\ \theta\in\Theta$$ a family of conjugate priors is made of distributions with densities $$\pi(\theta|x_0,\lambda)\propto \exp\{\theta^\prime x_0-\lambda\tau(\theta)\}\quad x_0\in\mathcal X_0,\ \lambda\in\Lambda$$ with respect to an arbitrary dominating measure $$\text d\nu$$ on $$\Theta$$, provided

1. $$\mathcal X_0+\mathcal X\subset\mathcal X_0$$
2. $$\Lambda+\mathbb N \subset \Lambda$$
3. $$\int_\Theta \exp\{\theta^\prime x_0-\lambda\tau(\theta)\}\,\text d\nu(\theta) <\infty$$ for all $$x_0\in\mathcal X_0,\ \lambda\in\Lambda$$.

It clearly defines an exponential family with sufficient statistic $$(\theta,\tau(\theta))$$.

The reverse problem is to start from an exponential family density on $$\theta$$ $$\pi(\theta|\xi)\propto\exp\{T(\theta)^\prime\xi-\delta(\xi)\}\propto\exp\{T(\theta)^\prime\xi\}$$ and to identify a likelihood enjoying the above as a conjugate family. This means first breaking $$T(\theta)^\prime\xi$$ into $$\theta^\prime x_0-\lambda\tau(\theta)$$ and second finding a measure $$\text dμ$$ on $$\mathcal X$$ such that $$\int_{\mathcal X} \exp\{θ′x\}\text dμ(x)=\exp\{τ(θ)\}$$ This is an issue when considering for instance the Poisson distributions $$\pi(\theta|\xi)\propto\exp\{\theta\xi-\log(\theta!)\}\mathbb I_\mathbb N(\theta)$$ because $$\theta$$ and $$\xi$$ are unidimensional. One would thus need to embed the Poisson family into a two-dimensional family $$\pi(\theta|\xi,\lambda)\propto \exp\{\theta\xi-\lambda\log(\theta!)\}\mathbb I_\mathbb N(\theta)$$ and then check if there exist $$\lambda_0\in\mathbb R$$, a set $$\mathcal X\subset\mathbb R$$, and a measure $$\text d\mu$$ on $$\mathcal X$$such that $$\int_\mathcal X \exp\{\theta x\} \,\text d\mu(x)=\exp\{\lambda\log(\theta!)\}\quad \forall\theta\in\mathbb N$$

"If the likelihood function belongs to the exponential family, then a conjugate prior exists", not the other way around. For example, many popular discrete distributions come from exponential family and have conjugate priors, but are not conjugate priors for other distributions, because the parameters of most distributions are not discrete.

• It is unclear to me why restricting the support of the parameter in the prior (or the dominating measure) is a reason for dismissing this prior. It is an unusual setting but not a contradiction of the definition of conjugate families. Feb 1, 2022 at 14:08
• @Xi'an I meant it rather as an example than a contradiction.
– Tim
Feb 1, 2022 at 14:13