3
$\begingroup$

It seems like every resource that discusses exponential families simply defines the family of distributions, explains why it's useful and then derives some of its properties.

I have only seen one resource which seems to derive the exponential family through geometry. Specifically, "Differential Geometry and Statistics" by Murray and Rice. I don't have the background to understand this exposition, but I still want to understand where the exponentially family comes from.

How can one arrive at the form exponential family of distributions from first principles without prior knowledge of what it looks like?

I'd like to know what question does the form of the exponential family answer?

The answer I'm looking for would follow along these lines:

  1. What distributions have X,Y,Z properties?
  2. Writing down assumptions based on this question, we'd "solve" for the set of distributions satisfying these properties.
  3. Arrive at an equation describing the set defined in step one which happens to be algebraically equivalent to the exponential family of distributions.
Improve this question
$\endgroup$
10
  • 1
    $\begingroup$ From a practical point of view I would guess that people realized that presumably different distributions have astonishingly similar properties so that they searched for a 'common root cause' of these properties. Then they started to 'fuse' single pairs of these 'presumably different' distributions and figured out that the 'root cause' for the similarities is that they surprisingly arise from the same family just with different parameters. So they continued in that way and fused and fused until they arrived at the stage today: the exponential family. It is consistent in an important... $\endgroup$
    – Fabian Werner
    Commented Feb 11, 2019 at 21:54
  • 1
    $\begingroup$ I don't understand the motivation to write this all as a comment $\endgroup$
    – libby
    Commented Feb 11, 2019 at 22:15
  • 1
    $\begingroup$ It outgrew as I was going, sorry ;-) I often start with a comment in order to get a better understanding of the question... $\endgroup$
    – Fabian Werner
    Commented Feb 11, 2019 at 23:04
  • 2
    $\begingroup$ I gave one answer to a more specific version of this question at stats.stackexchange.com/questions/32103/…. It invokes the "Principle of Mathematical Laziness" in a maximum likelihood setting. It's not the only possible answer because your question is more general. $\endgroup$
    – whuber ♦
    Commented Feb 11, 2019 at 23:47
  • 1
    $\begingroup$ @whuber are you saying that a random variable is in the exponential family if and only it has a closed form solution of its MLE? $\endgroup$
    – libby
    Commented Feb 12, 2019 at 4:10

1 Answer 1

Reset to default
1
$\begingroup$

From a practical point of view I would guess that people realized that presumably different distributions have astonishingly similar properties so that they searched for a 'common root cause' of these properties. Then they started to 'fuse' single pairs of these 'presumably different' distributions and figured out that the 'root cause' for the similarities is that they surprisingly arise from the same family just with different parameters.

So they continued in that way and fused and fused until they arrived at the stage today: the exponential family.

It is consistent in an important sense: if I do not misinterpret the wikipedia page then it means that the family stays consistent under using the Bayesian framework, i.e. if one chooses the prior correctly then the posterior of a member of the exponential family is again a member of the exponential family...

Now this property also holds for smaller families so you might ask: why did people stop here, i.e. why is the exp. family the 'right thing' or: why don't we enlarge it? This hints at the other questions: They maximize the entropy w.r.t. a reference measure dH, i.e. take some measure dH (probably it needs to satisfy mild assumptions) and take some other random variables T_1,...,T_n (also satisfying some mild conditions like that they are integrable and so on) then what is the measure dF such that this crazy integral in 'Maximum entropy derivation' in the wikipedia page with the additional constraint that the expected value of the T_i w.r.t. that measure dF is some fixed value t_i is a member of the exp. family that you construct from this input data. These are the equations that you search for (they are not easy though!)...

Maximum Entropy means 'contains the most information' somewhat...

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.