In Generalized Linear Models the conditional distribution of the response variable has to belong to the exponential family. Why is this restriction important? What property would a regression model lose if we chose a distribution outside the exponential family?

  • 2
    $\begingroup$ sufficiency of the maximum likelihood estimate is one one thing you will lose. $\endgroup$
    – utobi
    Nov 19 '16 at 14:30
  • 1
    $\begingroup$ Theory and algorithms are easier ti develop inside the exponential family. But you can go outside it, and it is done nowadays---see the R VGAM package, doing just this. $\endgroup$ Nov 19 '16 at 14:36

Jaynes makes the argument that when you leave the exponential family, your estimators cease to be sufficient statistics. If a statistic is sufficient for a parameter then $\Pr(t|\theta)=\Pr(X|\theta)$. Implying that the information in $t$ is the same as in the sample $X$. Bayesian methods always use all the information in $X$. Non-Bayesian methods use a statistic. If that statistic contains the same information then the estimator will be no worse.

If the statistic is not sufficient, then it will be noisier than the Bayesian estimate. Bayesian estimators are always admissible statistics. If the distribution is not in the exponential family, then the Bayesian estimator will stochastically dominate it, hence the estimator will be inadmissible.

So if you do not make such an assumption, then you would be better off using a Bayesian model in all circumstances. If that were the case, why would you use an alternative?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.