What properties are "deep" is a very subjective issue! so the answer depends on your concept of "deep". But, if having conjugate priors is a "deep" property, in some sense, then that sense is mathematical and not statistical. The only reason that (some) statisticians are interested in conjugate priors is that they simplify some computations. But that is less important for each day that passes!
EDIT
Trying to answer @whuber comment below. First, an answer needs to ask more precisely what is a conjugate family of priors? It means a family that is closed under sampling, so, (for the given sampling model), the prior and posterior distributions belong to the same family. That is clearly true for the family of all distributions, but that interpretation leaves the question without content, so we need a more limited interpretation. Further, as pointed out by Diaconis & Ylvisaker, for the binomial model, if we let $h$ be a bounded positive function on $[0,1]$ and $f(p;\alpha,\beta)$ be the beta density then $h(p)f(p;\alpha,\beta)$ is a conjugate prior. It lacks some of the properties of the usual beta conjugate prior, but the family it generates is closed under sampling, so a conjugate prior. We don't get nice closed formulas, but we only need one numerical integration to get the normalizing constant.
Now, the usual beta prior density has one further important property: The posterior expectation is a linear function:
$$ \DeclareMathOperator{\E}{\mathbb{E}}
\E \left\{ \E (\theta \mid X=x)\right\} = ax+b
$$
(for some $a,b$). The corresponding property holds for the "usual" conjugate priors in exponential families, see Diaconis & Ylvisaker. So in these sense the usual conjugate families plays a role in Bayesian statistics similar to the Gauss-Markov theorem in classical statistics (see Role of Gauss-Markov Theorem in Linear Regression): it is a justification for linear methods.
There is also another viewpoint leading to the usual conjugate families. If we think of the prior information as representing information from some prior data (from the same sampling distribution family), then we could incorporate this information as a prior likelihood function. Then we could get a combined likelihood function by multiplying the prior likelihood with the data likelihood. We could instead choose to represent the prior data information via a prior distribution, the "usual" conjugate prior is the choice that gives a $\text{prior}\times\text{likelihood}$ proportional to the combined likelihood above. See https://en.wikipedia.org/wiki/Conjugate_prior where this interpretation is used to give prior data interpretations to the parameters in the (usual) conjugate families listed.
So, summarizing, the usual conjugate families in exponential families can be justified as priors leading to linear methods, or as priors coming from representing prior data. Hope this extended answer helps!
