Is there a limiting distribution if you keep taking conjugate priors within the exponential family? I'm assuming this is pretty commonly discussed but haven't found it yet.
For example, if you start with a binary (0-1 binomial), conjugate prior is beta distribution. Beta distribution has some un-named conjugate prior. You can take the conjugate prior of that.
Is there anything interesting coming of this? Anything useful? Looking for links and references mostly.
 A: No, there is no end to this process, and you always end up with a distribution having one additional parameter.
Copied from my answer here:
Let's say that you have a distribution $F$ in the exponential family with density
\begin{align}
\newcommand{\mbx}{\mathbf x}
\newcommand{\btheta}{\boldsymbol{\theta}}
f(\mbx \mid \btheta) &= \exp\bigl(\eta(\btheta) \cdot T(\mbx) - g(\btheta) + h(\mbx)\bigr)
\end{align}
Given independent realizations $\{x_1, x_2, \dotsc, x_n\}$ of $F$ (with unknown parameter $\theta$), then the distribution over $\theta$, $F'$, is the conjugate prior of $F$.  The density of $F'$ is
\begin{align}
f(\btheta \mid \boldsymbol\phi) = L(\btheta \mid \mbx_1, \dotsc, \mbx_n)
&= f(\mbx_1, \dotsc, \mbx_n \mid \btheta) \\\\
&\propto \prod_i f(\mbx_i\mid \btheta) \\\\
&= \textstyle\prod_i\exp\Bigl(\eta(\btheta) \cdot \textstyle T\left(\mbx_i\right) - g(\btheta) + h(\mbx_i)\Bigr) \\\\
&\propto \textstyle\prod_i\exp\Bigl(\eta(\btheta) \cdot \textstyle T\left(\mbx_i\right) - g(\btheta)\Bigr) \\\\
&= \textstyle\exp\Bigl(\eta(\btheta) \cdot \bigl(\textstyle\sum_iT\left(\mbx_i\right)\bigr) - ng(\btheta)\Bigr) \\\\
&= \exp\bigl(\eta'(\boldsymbol \phi) \cdot T'(\btheta)\bigr)
\end{align}
where
\begin{align}
\eta'(\boldsymbol\phi) &=
\begin{bmatrix}
\sum_iT_1(\mbx_i) \\\\
\vdots \\\\
\sum_iT_k(\mbx_i) \\\\
-\sum_i1
\end{bmatrix}
&
T'(\btheta) &=
\begin{bmatrix}
\eta_1(\btheta) \\\\
\vdots \\\\
\eta_k(\btheta) \\\\
g(\btheta)
\end{bmatrix}.
\end{align}
Thus, $F'$  is also in the exponential family ($T'$ replaced $\eta$ and $\eta'$ replaced $T$ since this distribution is over $\theta$ the parameter of the distribution over $x$.)
Interestingly, $\boldsymbol\phi$ has exactly one more parameter than $\btheta$ except in the rare case where  natural parameter $\phi_{k+1}$ is redundant, but such a distribution would be very weird (it would mean that the number of observations $\mbx$, that is, $n$, tells you nothing about $\btheta$.)
So, to answer your question, with each conjugate prior you get exactly one more hyperparameter.
