3 added 2690 characters in body
source | link

What properties are "deep" is a very subjective issue! so the answer depends on your concept of "deep". But, if having conjugate priors is a "deep" property, in some sense, then that sense is mathematical and not statistical. The only reason that (some) statisticians are interested in conjugate priors is that they simplify some computations. But that is less important for each day that passes!

 EDIT

Trying to answer @whuber comment below. First, an answer needs to ask more precisely what is a conjugate family of priors? It means a family that is closed under sampling, so, (for the given sampling model), the prior and posterior distributions belong to the same family. That is clearly true for the family of all distributions, but that interpretation leaves the question without content, so we need a more limited interpretation. Furteh, as pointed out by Diaconis & Ylvisaker, for the binomial model, if we let $h$ be a bounded positive function on $[0,1]$ and $f(p;\alpha,\beta)$ be the beta density then $h(p)f(p;\alpha,\beta)$ is a conjugate prior. It lacks some of the properties of the usual beta conjugate prior, but the family it generates is closed under sampling, so a conjugate prior. We dont get nice closed formulas, but we only need one numerical integration to get the normalizing constant.

Now, the usual beta prior density has one further important property: The posterior expectation is a linear function: $$ \DeclareMathOperator{\E}{\mathbb{E}} \E \left\{ \E (\theta \mid X=x)\right\} = ax+b $$ (for some $a,b$). The corresponding property holds for the "usual" conjugate priors in exponential families, see Diaconis & Ylvisaker. So in these sense the usual conjuage families plays a role in bayesian statistics similar to the Gauss-Markov theorem in classical statistics (see Role of Gauss-Markov Theorem in Linear Regression): it is a justification for linear methods.

There is also another viewpoint leading to the usual conjugate families. If we think of the prior information as representing information from some prior data (from the same sampling distribution family), then we could incorporate this information as a prior likelihood function. Then we could get a combines likelihood function by multiplying the prior likelihood with the data likelihood. We could instead choose to represent the prior data information via a prior distribution, the "usual" conjugate prior is the choice that gives a $\text{prior}\times\text{likelihood}$ proportional to the combined likelihood above. See https://en.wikipedia.org/wiki/Conjugate_prior where this interpretation is used to give prior data interpretations to the parameters in the (usual) conjugate families listed.

So, summarizing, the usual conjugate familes in exponential families can be justified as priors leading to linear methods, or as priors coming from representing prior data. Hope this extended answer helps!

What properties are "deep" is a very subjective issue! so the answer depends on your concept of "deep". But, if having conjugate priors is a "deep" property, in some sense, then that sense is mathematical and not statistical. The only reason that (some) statisticians are interested in conjugate priors is that they simplify some computations. But that is less important for each day that passes!

What properties are "deep" is a very subjective issue! so the answer depends on your concept of "deep". But, if having conjugate priors is a "deep" property, in some sense, then that sense is mathematical and not statistical. The only reason that (some) statisticians are interested in conjugate priors is that they simplify some computations. But that is less important for each day that passes!

 EDIT

Trying to answer @whuber comment below. First, an answer needs to ask more precisely what is a conjugate family of priors? It means a family that is closed under sampling, so, (for the given sampling model), the prior and posterior distributions belong to the same family. That is clearly true for the family of all distributions, but that interpretation leaves the question without content, so we need a more limited interpretation. Furteh, as pointed out by Diaconis & Ylvisaker, for the binomial model, if we let $h$ be a bounded positive function on $[0,1]$ and $f(p;\alpha,\beta)$ be the beta density then $h(p)f(p;\alpha,\beta)$ is a conjugate prior. It lacks some of the properties of the usual beta conjugate prior, but the family it generates is closed under sampling, so a conjugate prior. We dont get nice closed formulas, but we only need one numerical integration to get the normalizing constant.

Now, the usual beta prior density has one further important property: The posterior expectation is a linear function: $$ \DeclareMathOperator{\E}{\mathbb{E}} \E \left\{ \E (\theta \mid X=x)\right\} = ax+b $$ (for some $a,b$). The corresponding property holds for the "usual" conjugate priors in exponential families, see Diaconis & Ylvisaker. So in these sense the usual conjuage families plays a role in bayesian statistics similar to the Gauss-Markov theorem in classical statistics (see Role of Gauss-Markov Theorem in Linear Regression): it is a justification for linear methods.

There is also another viewpoint leading to the usual conjugate families. If we think of the prior information as representing information from some prior data (from the same sampling distribution family), then we could incorporate this information as a prior likelihood function. Then we could get a combines likelihood function by multiplying the prior likelihood with the data likelihood. We could instead choose to represent the prior data information via a prior distribution, the "usual" conjugate prior is the choice that gives a $\text{prior}\times\text{likelihood}$ proportional to the combined likelihood above. See https://en.wikipedia.org/wiki/Conjugate_prior where this interpretation is used to give prior data interpretations to the parameters in the (usual) conjugate families listed.

So, summarizing, the usual conjugate familes in exponential families can be justified as priors leading to linear methods, or as priors coming from representing prior data. Hope this extended answer helps!

2 light editing
source | link

What properties are "deep" is a very subjective issue! so the answer depends on your concept of "deep". But, if having conjugate priors is a "deep" property, in some sense, then that sense is mathematical and nlotnot statistical. The only reason that (some) statisticians are interetedinterested in conjugate priors is that they simplify some computations. But that is less important for each day that passes!

What properties are "deep" is a very subjective issue! so the answer depends on your concept of "deep". But, if having conjugate priors is a "deep" property, in some sense, then that sense is mathematical and nlot statistical. The only reason that (some) statisticians are intereted in conjugate priors is that they simplify some computations. But that is less important for each day that passes!

What properties are "deep" is a very subjective issue! so the answer depends on your concept of "deep". But, if having conjugate priors is a "deep" property, in some sense, then that sense is mathematical and not statistical. The only reason that (some) statisticians are interested in conjugate priors is that they simplify some computations. But that is less important for each day that passes!

1
source | link

What properties are "deep" is a very subjective issue! so the answer depends on your concept of "deep". But, if having conjugate priors is a "deep" property, in some sense, then that sense is mathematical and nlot statistical. The only reason that (some) statisticians are intereted in conjugate priors is that they simplify some computations. But that is less important for each day that passes!