Do conjugate priors just lead to a posterior that is a modification of the parameters of the prior?

Question

We know, for example, that the conjugate relationship between the classic beta-binomial is as follows:

\begin{align} y &∼ \mathcal{Bin}(n,\ θ) \\ θ &∼ \mathcal{Beta}(α,\ β) \\ θ|y &∼ \mathcal{Beta}(y + α,\ n − y + β) \end{align}

Notice how the posterior is just a change in parameterization compared to the prior. Is the posterior of conjugate priors just a change in parameters?

Answer 1

This question is actually somewhat subtle, and it brings to attention an interesting quirk of usage that I hadn't noticed before.

For every practical definition of conjugate distributions that I'm familiar with, it is the case that the posterior of a model using a conjugate prior is a modified form of the prior. The wikipedia definition follows the "practicality" (convenience) convention, for example:

In Bayesian probability theory, if the posterior distributions $p(\theta|x)$ are in the same family as the prior probability distribution $p(\theta)$, the prior and posterior are then called conjugate distributions, and the prior is called a conjugate prior for the likelihood function

However, a distinction can be found in the formal definition of conjugacy in Gelman's Bayesian Data Analysis, 3rd edition, p. 35:

If $\mathcal{F}$ is a class of sampling distributions $p(y|\theta)$ and $\mathcal{P}$ is a class of prior distributions for $\theta$, then the class $\mathcal{P}$ is conjugate for $\mathcal{F}$ if $$ p(\theta|y)\in\mathcal{P}\forall p(\cdot|\theta)\in\mathcal{F} \text{ and } p(\cdot)\in\mathcal{P}. $$ This definition is formally vague since if we choose $\mathcal{P}$ as the class of all distributions, then $\mathcal{P}$ is always conjugate no matter what class of sampling distribution is used.

Obviously the construction in the final sentence has little practical utility: if all distributions are conjugate, then the distinction between conjugate and non-conjugate distributions is meaningless. Instead, it is common to take $\mathcal{P}$ to be the set of all densities having the same functional form of the likelihood, giving rise to the practical convenience properties of conjugacy, namely that the posterior is the form of the prior.