How many natural parameters are really in the exponential family conjugate prior?

Question

The exponential family with natural parameter $\theta$ can be written $$ p(x|\theta)=h_\ell(x)\exp(\theta^Tt(x)-a_\ell(\theta)) $$ with conjugate prior $$ p(\theta|\lambda)=h_c(\theta)\exp(\lambda_1^T\theta+\lambda_2(-a_\ell(\theta))-a_c(\lambda)), $$ where the natural parameter $\lambda=(\lambda_1,\lambda_2)$ clearly has dimension $\dim(\theta)+1$ and the sufficient statistics can be written $(\theta,-a_\ell(\theta))$.

Consider the case for univariate normally distributed data with unknown mean $\mu$ and unknown precision $\tau$. Then the natural parameter $\theta$ has dimension 2. We know its conjugate prior is normal-gamma. The Wikipedia page for the exponential family states that the normal-gamma distribution has 4 natural parameters. However, given the above definition for the conjugate prior, I would expect $\dim(\theta)+1=3$ natural parameters. How can this definition for the normal-gamma distribution fit into the above conjugate prior expression? Is it not generally true that the conjugate prior has $\dim(\theta)+1$ parameters?

Answer 1

It is a correct remark that a conjugate family of priors for an exponential family of distributions with a two-dimensional parameter $(\theta_1,\theta_2)$ can be defined by three parameters and hence that for a Normal family of distributions $$f(x|\theta_1,\theta_2)\propto \exp\left\{\frac{-x^2}{2\theta_2}+\frac{x\theta_1}{\theta_2}-\frac{\theta_1^2}{2\theta_2}-\frac{\log(\theta_2)}{2}\right\}$$ a conjugate family of priors is defined by $$\pi(\theta_1,\theta_2|\alpha_1,\alpha_2,\lambda)\propto \exp\left\{\frac{-\alpha_1}{2\theta_2}+\frac{\alpha_2\theta_1}{\theta_2}-\lambda\frac{\theta_1^2}{2\theta_2}-\lambda\frac{\log(\theta_2)}{2}\right\}$$ However it is also possible to defined conjugate priors with more hyperparameters in the sense that priors and posteriors belong to the same family.

In our book Bayesian Essentials with R, we do resort to the four parameter version:

We now consider the general case of an iid sample $\mathscr{D}_n=(x_1,\ldots,x_n)$ from the normal distribution $\mathscr{N}(\mu,\sigma^2)$ and $\theta=(\mu,\sigma^2)$. This setting also allows a conjugate prior since the normal distribution remains an exponential family when both parameters are unknown. It is of the form $$ (\sigma^2)^{-\lambda_\sigma-3/2}\,\exp\left\{-\left(\lambda_\mu(\mu-\xi)^2+\alpha\right)/2\sigma^2\right\} $$ since \begin{eqnarray}\label{eq:conjunor} \pi((\mu,\sigma^2)|\mathscr{D}_n) & \propto & (\sigma^2)^{-\lambda_\sigma-3/2}\, \exp\left\{-\left(\lambda_\mu (\mu-\xi)^2 + \alpha \right)/2\sigma^2\right\}\nonumber\\ && \times (\sigma^2)^{-n/2}\,\exp \left\{-\left(n(\mu-\overline{x})^2 + s_x^2 \right)/2\sigma^2\right\} \\ &\propto& (\sigma^2)^{-\lambda_\sigma(\mathscr{D}_n)}\exp\left\{-\left(\lambda_\mu(\mathscr{D}_n) (\mu-\xi(\mathscr{D}_n))^2+\alpha(\mathscr{D}_n)\right)/2\sigma^2\right\}\,,\nonumber \end{eqnarray} where $s_x^2 = \sum_{i=1}^n (x_i-\overline{x})^2$. Therefore, the conjugate prior on $\theta$ is the product of an inverse gamma distribution on $\sigma^2$, $\mathscr{IG}(\lambda_\sigma,\alpha/2)$, and, conditionally on $\sigma^2$, a normal distribution on $\mu$, $\mathscr{N} (\xi,\sigma^2/\lambda_\mu)$.

but there is not particular reason for doing so, except if a different degree of prior precision ($\lambda_\mu\ne\lambda_\sigma$) is available on $\mu$ and on $\sigma$.