In Generalized Linear Models the conditional distribution of the response variable has to belong to the exponential family. Why is this restriction important? What property would a regression model lose if we chose a distribution outside the exponential family?
$\begingroup$
$\endgroup$
2
-
4$\begingroup$ sufficiency of the maximum likelihood estimate is one one thing you will lose. $\endgroup$– utobiCommented Nov 19, 2016 at 14:30
-
1$\begingroup$ Theory and algorithms are easier ti develop inside the exponential family. But you can go outside it, and it is done nowadays---see the R VGAM package, doing just this. $\endgroup$– kjetil b halvorsen ♦Commented Nov 19, 2016 at 14:36
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Jaynes makes the argument that when you leave the exponential family, your estimators cease to be sufficient statistics. If a statistic is sufficient for a parameter then $\Pr(t|\theta)=\Pr(X|\theta)$. Implying that the information in $t$ is the same as in the sample $X$. Bayesian methods always use all the information in $X$. Non-Bayesian methods use a statistic. If that statistic contains the same information then the estimator will be no worse.
If the statistic is not sufficient, then it will be noisier than the Bayesian estimate. Bayesian estimators are always admissible statistics. If the distribution is not in the exponential family, then the Bayesian estimator will stochastically dominate it, hence the estimator will be inadmissible.
So if you do not make such an assumption, then you would be better off using a Bayesian model in all circumstances. If that were the case, why would you use an alternative?
-
$\begingroup$ @ Dave Harris: great answer! I had some questions about your answer, I posted them over here: stats.stackexchange.com/questions/552707/… ...maybe you can take a look at it later if you have time? thank you! $\endgroup$ Commented Nov 18, 2021 at 8:03
-
$\begingroup$ Who is Jaynes? Also, can you justify your sentence "when you leave the exponential family your estimators cease to be sufficient"? It is clear that MLEs are sufficient for exponential family. Can you really say that ANY estimator will not be sufficient, whenever you leave the exponential family? It is a strange sentence, "estimator" and "random component" are different animals that might not even be related! Are we strictly talking about MLEs here? $\endgroup$– Car LozCommented Jan 29 at 20:15