Having a conjugate prior: Deep property or mathematical accident? Some distributions have conjugate priors and some do not.  Is this distinction just an accident? That is, you do the math, and it works out one way or the other, but it does not really tell you anything important about the distribution except for the fact itself?
Or does the presence or absence of a conjugate prior reflect some deeper property of a distribution? Do distributions with conjugate priors share some other interesting property or properties that other distributions lack that causes those distributions, and not the others, to have a conjugate prior?
 A: I am very new to Bayesian statistics, but it seems to me that all of these distributions (and if not all of them then at least those that are useful) share the property that they are described by some limited metric about the observations that define them. I.e., for a normal distribution, you don't need to know every detail about every observation, just their total count and sum.
To put it another way, assuming you already know the class/family of distribution, then the distribution has strictly lower information entropy than the observations that resulted in it.
Does this seem trivial, or is it kind of what you're looking for?
A: It is not by accident. Here you shall find a brief a very nice review on conjugate priors. Concretely, it mentions that if there exist a set of sufficient statistics of fixed dimension for the given likelihood function, then you can construct a conjugate prior to it. Having a set of sufficient statistics means that you can factorize the likelihood in a form that lets you estimate the parameters in a computationally efficient way.
Besides that, having conjugate priors is not only computationally convenient. It also provides smoothing and allows to work with very little samples or no previous samples, which is necessary for problems like decision making, in cases where you have very little evidence.
A: What properties are "deep" is a very subjective issue! so the answer depends on your concept of "deep". But, if having conjugate priors is a "deep" property, in some sense, then that sense is mathematical and not statistical. The only reason that (some) statisticians are interested in conjugate priors is that they simplify some computations. But that is less important for each day that passes!
 EDIT

Trying to answer @whuber comment below. First, an answer needs to ask more precisely what is a conjugate family of priors? It means a family that is closed under sampling, so, (for the given sampling model), the prior and posterior distributions belong to the same family.  That is clearly true for the family of all distributions, but that interpretation leaves the question without content, so we need a more limited interpretation. Further, as pointed out by Diaconis & Ylvisaker, for the binomial model, if we let $h$ be a bounded positive function on $[0,1]$ and $f(p;\alpha,\beta)$ be the beta density then $h(p)f(p;\alpha,\beta)$ is a conjugate prior.  It lacks some of the properties of the usual beta conjugate prior, but the family it generates is closed under sampling, so a conjugate prior.  We don't get nice closed formulas, but we only need one numerical integration to get the normalizing constant.
Now, the usual beta prior density has one further important property: The posterior expectation is a linear function:
$$ \DeclareMathOperator{\E}{\mathbb{E}}
   \E \left\{ \E (\theta \mid X=x)\right\} = ax+b
$$
(for some $a,b$).  The corresponding property holds for the "usual" conjugate priors in exponential families, see Diaconis & Ylvisaker.  So in these sense the usual conjugate families plays a role in Bayesian statistics similar to the Gauss-Markov theorem in classical statistics (see Role of Gauss-Markov Theorem in Linear Regression): it is a justification for linear methods.
There is also another viewpoint leading to the usual conjugate families. If we think of the prior information as representing information from some prior data (from the same sampling distribution family), then we could incorporate this information as a prior likelihood function.  Then we could get a combined likelihood function by multiplying the prior likelihood with the data likelihood.  We could instead choose to represent the prior data information via a prior distribution, the "usual" conjugate prior is the choice that gives a $\text{prior}\times\text{likelihood}$ proportional to the combined likelihood above. See https://en.wikipedia.org/wiki/Conjugate_prior  where this interpretation is used to give prior data interpretations to the parameters in the (usual) conjugate families listed.
So, summarizing, the usual conjugate families in exponential families can be justified as priors leading to linear methods, or as priors coming from representing prior data. Hope this extended answer helps!
