# How many natural parameters are really in the exponential family conjugate prior?

The exponential family with natural parameter $\theta$ can be written $$p(x|\theta)=h_\ell(x)\exp(\theta^Tt(x)-a_\ell(\theta))$$ with conjugate prior $$p(\theta|\lambda)=h_c(\theta)\exp(\lambda_1^T\theta+\lambda_2(-a_\ell(\theta))-a_c(\lambda)),$$ where the natural parameter $\lambda=(\lambda_1,\lambda_2)$ clearly has dimension $\dim(\theta)+1$ and the sufficient statistics can be written $(\theta,-a_\ell(\theta))$.

Consider the case for univariate normally distributed data with unknown mean $\mu$ and unknown precision $\tau$. Then the natural parameter $\theta$ has dimension 2. We know its conjugate prior is normal-gamma. The Wikipedia page for the exponential family states that the normal-gamma distribution has 4 natural parameters. However, given the above definition for the conjugate prior, I would expect $\dim(\theta)+1=3$ natural parameters. How can this definition for the normal-gamma distribution fit into the above conjugate prior expression? Is it not generally true that the conjugate prior has $\dim(\theta)+1$ parameters?

• @Xi’an has given a completely satisfactory answer, but just to elaborate a bit, there is more than one “conjugate family” one can have generally speaking. In some sense, we could define a useless conjugate family: the set of all distributions for $\theta$ with a density. Trivially this family is conjugate because any prior in the family will result in a posterior in the family. The only requirement for conjugate families is that you start and end in the family (someone might add another requirement that you can index by a finite fixed-dimension parameter, but that isn’t required typically). – guy Sep 2 at 4:57
• @guy: (+1) yes there is an infinite collection of conjugate families, actually it starts with the choice of the dominating measure! – Xi'an Sep 2 at 9:20
• @guy This is an interesting but troubling comment! The engineer in me really hopes for “the” singular conjugate prior for a particular exponential family member. Is there some criteria by which one might “design” a particular conjugate prior for a given member of the exponential family? There must be an established way to exploit the flexibility of the conjugate family. Sticking with the example of normally distributed data, why is it that by far, the most typical instantiation of a conjugate prior is the 4-parameter normal-gamma? I have a feeling it isn’t purely by chance... – scherm Sep 3 at 19:58
• @scherm You can define a conjugate prior for an exponential family more-or-less by just defining the family to be an exponential tilt of an arbitrary baseline prior. This is related to @Xi’an comment about the usual conjugate prior also depending on the dominating measure. The issue is that you are not guaranteed that this family will be useful like the normal-gamma. But you can (for example, tilting a gamma distribution to get a conjugate family for the variance in a normal model leads to generalized inverse Gaussian priors, which are conjugate to the normal model with unknown variance). – guy Sep 3 at 21:57
• You should not seek "the" conjugate prior, as this contradicts the Bayesian notion of allowing for any prior in a Bayesian analysis. If there were a singular prior to use by default, there would be no Bayesian analysis. Even the limiting improper priors defined by letting the hyperparameters get to infinity are not unique, e.g., do not produce the Jeffreys prior all the time. – Xi'an Sep 4 at 3:10

It is a correct remark that a conjugate family of priors for an exponential family of distributions with a two-dimensional parameter $(\theta_1,\theta_2)$ can be defined by three parameters and hence that for a Normal family of distributions $$f(x|\theta_1,\theta_2)\propto \exp\left\{\frac{-x^2}{2\theta_2}+\frac{x\theta_1}{\theta_2}-\frac{\theta_1^2}{2\theta_2}-\frac{\log(\theta_2)}{2}\right\}$$ a conjugate family of priors is defined by $$\pi(\theta_1,\theta_2|\alpha_1,\alpha_2,\lambda)\propto \exp\left\{\frac{-\alpha_1}{2\theta_2}+\frac{\alpha_2\theta_1}{\theta_2}-\lambda\frac{\theta_1^2}{2\theta_2}-\lambda\frac{\log(\theta_2)}{2}\right\}$$ However it is also possible to defined conjugate priors with more hyperparameters in the sense that priors and posteriors belong to the same family.
We now consider the general case of an iid sample $\mathscr{D}_n=(x_1,\ldots,x_n)$ from the normal distribution $\mathscr{N}(\mu,\sigma^2)$ and $\theta=(\mu,\sigma^2)$. This setting also allows a conjugate prior since the normal distribution remains an exponential family when both parameters are unknown. It is of the form $$(\sigma^2)^{-\lambda_\sigma-3/2}\,\exp\left\{-\left(\lambda_\mu(\mu-\xi)^2+\alpha\right)/2\sigma^2\right\}$$ since \begin{eqnarray}\label{eq:conjunor} \pi((\mu,\sigma^2)|\mathscr{D}_n) & \propto & (\sigma^2)^{-\lambda_\sigma-3/2}\, \exp\left\{-\left(\lambda_\mu (\mu-\xi)^2 + \alpha \right)/2\sigma^2\right\}\nonumber\\ && \times (\sigma^2)^{-n/2}\,\exp \left\{-\left(n(\mu-\overline{x})^2 + s_x^2 \right)/2\sigma^2\right\} \\ &\propto& (\sigma^2)^{-\lambda_\sigma(\mathscr{D}_n)}\exp\left\{-\left(\lambda_\mu(\mathscr{D}_n) (\mu-\xi(\mathscr{D}_n))^2+\alpha(\mathscr{D}_n)\right)/2\sigma^2\right\}\,,\nonumber \end{eqnarray} where $s_x^2 = \sum_{i=1}^n (x_i-\overline{x})^2$. Therefore, the conjugate prior on $\theta$ is the product of an inverse gamma distribution on $\sigma^2$, $\mathscr{IG}(\lambda_\sigma,\alpha/2)$, and, conditionally on $\sigma^2$, a normal distribution on $\mu$, $\mathscr{N} (\xi,\sigma^2/\lambda_\mu)$.
but there is not particular reason for doing so, except if a different degree of prior precision ($\lambda_\mu\ne\lambda_\sigma$) is available on $\mu$ and on $\sigma$.