It seems like every resource that discusses exponential families simply defines the family of distributions, explains why it's useful and then derives some of its properties.

I have only seen one resource which seems to derive the exponential family through geometry. Specifically, "Differential Geometry and Statistics" by Murray and Rice. I don't have the background to understand this exposition, but I still want to understand where the exponentially family comes from.

How can one arrive at the form exponential family of distributions from first principles without prior knowledge of what it looks like?

I'd like to know what question does the form of the exponential family answer?

The answer I'm looking for would follow along these lines:

  1. What distributions have X,Y,Z properties?
  2. Writing down assumptions based on this question, we'd "solve" for the set of distributions satisfying these properties.
  3. Arrive at an equation describing the set defined in step one which happens to be algebraically equivalent to the exponential family of distributions.
  • $\begingroup$ From a practical point of view I would guess that people realized that presumably different distributions have astonishingly similar properties so that they searched for a 'common root cause' of these properties. Then they started to 'fuse' single pairs of these 'presumably different' distributions and figured out that the 'root cause' for the similarities is that they surprisingly arise from the same family just with different parameters. So they continued in that way and fused and fused until they arrived at the stage today: the exponential family. It is consistent in an important... $\endgroup$ – Fabian Werner Feb 11 at 21:54
  • $\begingroup$ sense: if I do not misinterpret the wikipedia page then it means that the family stays consistent under using the Bayesian framework, i.e. if one chooses the prior correctly then the posterior of a member of the exponential family is again a member of the exponential family... Now this property also holds for smaller families so you might ask: why did people stop here, i.e. why is the exp. family the 'right thing' or: why don't we enlarge it? This hints at the other questions: They maximize the entropy w.r.t. a reference measure dH, i.e. take some measure dH (probably it needs to satisfy... $\endgroup$ – Fabian Werner Feb 11 at 21:59
  • $\begingroup$ mild assumptions) and take some other random variables T_1,...,T_n (also satisfying some mild conditions like that they are integrable and so on) then what is the measure dF such that this crazy shit integral in 'Maximum entropy derivation' in the wikipedia page with the additional constraint that the expected value of the T_i w.r.t. that measure dF is some fixed value t_i is a member of the exp. family that you construct from this input data. These are the equations that you search for (they are not easy though!)... Maximum Entropy means 'contains the most information' somewhat... $\endgroup$ – Fabian Werner Feb 11 at 22:03
  • $\begingroup$ I don't understand the motivation to write this all as a comment $\endgroup$ – libby Feb 11 at 22:15
  • 1
    $\begingroup$ I gave one answer to a more specific version of this question at stats.stackexchange.com/questions/32103/…. It invokes the "Principle of Mathematical Laziness" in a maximum likelihood setting. It's not the only possible answer because your question is more general. $\endgroup$ – whuber Feb 11 at 23:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.