Questions tagged [exponential-family]

A set of distributions (eg, normal, $\chi^2$, Poisson, etc) that share a specific form. Many of the distributions in the exponential family are standard, workhorse distributions in statistics, w/ convenient statistical properties.

Filter by
Sorted by
Tagged with
36 votes
4 answers
4k views

Why is the exponential family so important in statistics?

Why is the exponential family so important in statistics? I was recently reading about the exponential family within statistics. As far as I understand, the exponential family refers to any ...
  • 6,028
29 votes
1 answer
2k views

When if ever is a median statistic a sufficient statistic?

I came across a casual remark on The Chemical Statistician that a sample median could often be a choice for a sufficient statistic but, besides the obvious case of one or two observations where it ...
  • 94.5k
21 votes
3 answers
4k views

Why doesn't the exponential family include all distributions?

I am reading the book: Bishop, Pattern Recognition and Machine Learning (2006) which defines the exponential family as distributions of the form (Eq. 2.194): $$ p(\mathbf x|\boldsymbol \eta) = h(\...
  • 3,368
21 votes
2 answers
2k views

Advantages of the Exponential Family: why should we study it and use it?

So here I am studying inference. I would like that someone could enumerate the advantages of the exponential family. By exponential family, I mean the distributions which are given as \begin{align*} f(...
's user avatar
21 votes
0 answers
909 views

Is there a general expression for ancillary statistics in exponential families?

An i.i.d sample $X_1,\dots,X_n$ from a scale family with c.d.f. $F(\frac{x}{\sigma})$ has $S(X)$ as an ancillary statistic if $S(X)$ depends on the sample only through $\frac{X_1}{X_n},\cdots,\frac{X_{...
  • 2,300
19 votes
2 answers
5k views

Does log likelihood in GLM have guaranteed convergence to global maxima?

My questions are: Are generalized linear models (GLMs) guaranteed to converge to a global maximum? If so, why? Furthermore, what constraints are there on the link function to insure convexity? My ...
17 votes
2 answers
6k views

Kullback–Leibler divergence between two gamma distributions

Choosing to parameterize the gamma distribution $\Gamma(b,c)$ by the pdf $g(x;b,c) = \frac{1}{\Gamma(c)}\frac{x^{c-1}}{b^c}e^{-x/b}$ The Kullback-Leibler divergence between $\Gamma(b_q,c_q)$ and $\...
16 votes
3 answers
10k views

Poisson is to exponential as Gamma-Poisson is to what?

A Poisson distribution can measure events per unit time, and the parameter is $\lambda$. The exponential distribution measures the time until next event, with the parameter $\frac{1}{\lambda}$. One ...
  • 3,363
16 votes
3 answers
6k views

Definition of family of a distribution?

Does a family of a distribution have a different definition for statistics than in other disciplines? In general, a family of curves is a set of curves, each of which is given by a function or ...
  • 11.9k
16 votes
2 answers
279 views

Derivation of normalizing transform for GLMs

$\newcommand{\E}{\mathbb{E}}$How is the $A(\cdot) = \displaystyle\int\frac{du}{V^{1/3}(\mu)}$ normalizing transform for the exponential family derived? More specifically: I tried to follow the ...
  • 479
14 votes
4 answers
3k views

What is the rationale behind the exponential family of distributions?

From elementary probability course, the probability distributions such as Gaussian, Poisson or exponential all have a good motivation. After staring at the formula of the exponential family ...
13 votes
1 answer
10k views

ML estimate of exponential distribution (with censored data)

In Survival Analysis, you assume the survival time of a r.v. $X_i$ to be exponentially distributed. Considering now that I have $x_1,\dots,x_n$ "outcomes" of i.i.d r.v.'s $X_i$. Only some proportion ...
13 votes
1 answer
998 views

Aside from the exponential family, where else can conjugate priors come from?

Do all conjugate priors have to come from the exponential family? If not, what other families are known to have/produce conjugate priors?
  • 3,588
12 votes
1 answer
1k views

Do the mean and the variance always exist for exponential family distributions?

Assume a scalar random variable $X$ belongs to a vector-parameter exponential family with p.d.f. $$ f_X(x|\boldsymbol \theta) = h(x) \exp\left(\sum_{i=1}^s \eta_i({\boldsymbol \theta}) T_i(x) - A({\...
  • 345
12 votes
1 answer
1k views

Does a canonical link function always exist for a Generalized Linear Model (GLM)?

In GLM, assuming a scalar $Y$ and $\theta$ for the underlying distribution with p.d.f. $$f_Y(y | \theta, \tau) = h(y,\tau) \exp{\left(\frac{\theta y - A(\theta)}{d(\tau)} \right)}$$ It can be shown ...
  • 345
11 votes
1 answer
7k views

Why is the continuous uniform distribution not an exponential family?

In class I've been given that a family of distributions $\{P_{\theta} : \theta \in \Theta\}, \Theta \in \mathbb{R}^{k} $ is an exponential family if there exits real-valued functions $\eta_{1}, ... , \...
11 votes
1 answer
3k views

How to derive the conjugate prior of an exponential family distribution

I am trying to derive the conjugate prior of the univariate Gaussian distribution over both the mean and the precision. I know that the prior I'm looking for is the normal-gamma distribution, but the ...
  • 111
11 votes
1 answer
878 views

Gaussian Like distribution with higher order moments

For the Gaussian distribution with unknown mean and variance, the sufficient statistics in the standard exponential family form is $T(x)=(x,x^2)$. I have a distribution that has $T(x)=(x,x^2,...,x^{2N}...
  • 523
10 votes
1 answer
3k views

Exponential Family: Observed vs. Expected Sufficient Statistics

My question arises from reading reading Minka's "Estimating a Dirichlet Distribution", which states the following without proof in the context of deriving a maximum-likelihood estimator for a ...
10 votes
2 answers
976 views

Find the joint distribution of $X_1$ and $\sum_{i=1}^n X_i$

This question is from Robert Hogg's Introduction to Mathematical Statistics 6th version question 7.6.7. The problem is : Let a random sample of size $n$ be taken from a distribution with the pdf $...
  • 4,537
10 votes
1 answer
2k views

Find UMVUE of $\frac{1}{\theta}$ where $f_X(x\mid\theta) =\theta(1 +x)^{−(1+\theta)}I_{(0,\infty)}(x)$

Let $X_1, X_2, . . . , X_n$ be iid random variables having pdf $$f_X(x\mid\theta) =\theta(1 +x)^{−(1+\theta)}I_{(0,\infty)}(x)$$ where $\theta >0$. Give the UMVUE of $\frac{1}{\theta}$ ...
  • 1,570
10 votes
1 answer
1k views

Unbiased estimator with minimum variance for $1/\theta$

Let$ X_1, ...,X_n$ be a random sample feom a distribution $Geometric(\theta)$ for $0<\theta<1$. I.e, $$p_{\theta}(x)=\theta(1-\theta)^{x-1} I_{\{1,2,...\}}(x)$$ Find the unbiased estimator ...
  • 1,138
9 votes
1 answer
18k views

How should you express a negative binomial distribution in an exponential family form?

When $X$ is $X_1$,...$X_n$, how do you express the following negative binomial distribution in an exponential family form? $$ f(k;r,p)\equiv\text{Pr}(X=k)=\binom{k+r-1}{k}(1-p)^{r}p^k~~~~~\text{for}~...
  • 171
9 votes
1 answer
1k views

What is meant by the term 'exponential family'? Why it is named so?

I have come across the term exponential family. The Bernoulli, Gaussian and many more distributions come under this exponential family. What would be the commonalities between them?
  • 527
9 votes
2 answers
2k views

Why is the mixtures of conjugate priors important?

I have questions about the mixture of conjugate priors. I learned and saw the mixture of conjugate priors a couple of times when I am learning bayesian. I am wondering why this theorem is such ...
9 votes
2 answers
954 views

A stochastically increasing exponential family for which $\lim_{\theta\rightarrow\inf\Theta}\mbox{P}_\theta(X\leq x)\neq 1$

Question A little something that I've been wondering about for a while: Let $P_\theta$ be a stochastically increasing (one-parameter) exponential family on the sample space $\mathcal{X}$ with $\...
  • 10.3k
9 votes
2 answers
2k views

Checking whether a density is exponential family

Trying to prove that this doesn't belong to exponential family. $f(y|a)=4\frac{(y+a)}{(1+4a)} ; 0 < y < 1 , a>0$ Here is my approach: $$f(y|a) = 4(y+a)e^{-log(1+4a)}$$ $$f(y|a) = (4y)(1+\...
  • 821
9 votes
1 answer
402 views

Does a sufficient statistic imply the existence of a conjugate prior?

In the comments on this answer, user Scortchi asks: So iff there's a sufficient statistic of constant dimension, there's a conjugate prior? As far as I know this didn't get a complete answer, so I'm ...
  • 355
9 votes
1 answer
199 views

Why is the EM algorithm well suited for exponential families?

I've been brushing up on the EM algorithm, and while I feel like I understand the basics, I keep seeing the claim made (e.g. here, here, among several others) that EM works particularly well for ...
  • 91
9 votes
1 answer
2k views

Is the negative binomial not expressible as in the exponential family if there are 2 unknowns?

I had a homework assignment to express the negative binomial distribution as an exponential family of distributions given that the dispersion parameter was a known constant. This was fairly easy, but ...
  • 91
8 votes
1 answer
1k views

Is the t distribution a member of the exponential family?

From what I understand, the exponential family is defined as $$f(y;\theta,\phi) = \exp\left(\frac{y\theta - b(\theta)}{a(\phi)}+c(y,\phi)\right) $$ I've read (but not seen shown anywhere), that the ...
8 votes
2 answers
5k views

Finding UMVUE of $\theta e^{-\theta}$ where $X_i\sim\text{Pois}(\theta)$

Suppose $X_1, X_2, . . . , X_n$ are i.i.d Poisson ($\theta$) random variables, where $\theta\in(0,\infty)$. Give the UMVUE of $\theta e^{-\theta}$ I found a similar problem here. I have that the ...
  • 1,570
8 votes
2 answers
1k views

Sufficient Statistic for non-exponential family distribution

Question: Let $X_1,X_2,\ldots,X_n$ be an iid sample from $N(\theta , 4 \theta^2 )$. I want to show that this model is not a member of the exponential family and to find a sufficient statistic for $\...
  • 1,146
8 votes
1 answer
2k views

Conditional distribution for Exponential family

We have a random variable $X$ that belongs to the exponential family with p.d.f. $$ P_X(x|\boldsymbol \theta) = h(x) \exp\left(\eta({\boldsymbol \theta}) . T(x) - A({\boldsymbol \theta}) \right) $$ ...
  • 439
8 votes
1 answer
4k views

Why do we assume the exponential family in the GLM context?

When I first learned about Generalized Linear Models I thought that the assumption that the dependent variable follows some distribution from the exponential family was made to simplify calculations. ...
8 votes
2 answers
2k views

Role of base measure in exponential family

An exponential family distribution $p$ in the canonical form can be written as $p(x|\theta) = h(x)\exp(\theta^\top T(x) - A(\theta))$ where $A(\theta)$ is the log partition function, $T(x)$ is the ...
  • 2,043
8 votes
1 answer
231 views

Support vector machines (SVMs) are the zero temperature limit of logistic regression?

I was had a quick discussion recently with a knowledgeable friend who mentioned that SVMs are the zero temperature limit of logistic regression. The rationale involved marginal polytopes and fenchel ...
  • 601
8 votes
1 answer
237 views

The distribution of a sufficient statistic

If I understand correctly, a distribution in the exponential family... $$\underline X\sim f_{\underline\theta}(\underline x) = exp\{\sum\limits_{i}\eta_i(\underline\theta)T_i(\underline x)-B(\...
  • 2,104
8 votes
1 answer
524 views

Generalized Linear Models - What's special about the exponential family?

In Generalized Linear Models the conditional distribution of the response variable has to belong to the exponential family. Why is this restriction important? What property would a regression model ...
  • 975
8 votes
0 answers
2k views

"weight" input in glm.nb function in R. How exactly does the weight affect the likelihood?

I would like to understand how the weight argument of glm.nb is affecting the likelihood function. I understand that glm.nb find ...
7 votes
3 answers
3k views

Which parameter should be considered as "scale" parameter for Gamma distribution?

From Wikipedia and probably all statistics textbooks, we know that in the density of a Gamma random variable $$f(x; k, \theta) = \frac{1}{\Gamma(k)\theta^k}x^{k - 1}e^{-\frac{x}{\theta}}, \quad x > ...
  • 5,102
7 votes
1 answer
185 views

Is $\mathrm{binomial}(n, p)$ family both full and curved for $n$ fixed?

Let $n$ be a fixed positive integer. The binomial$(n, p)$ family is given by $$f(x|p)=\tbinom{n}{x}p^x(1-p)^{n-x}\tag{1}.$$ We may rewrite (1) as $$f(x|p)=\tbinom{n}{x}(1-p)^n\exp\left[x\log\frac{p}{1-...
7 votes
1 answer
252 views

Name and interpretation of "$h(x)$" in exponential family

The exponential family is defined (in many sources) as: $$p(x | \theta) = h(x) \exp\{\theta^TT(x) - A(\theta)\}$$ where: $T(x)$ is a sufficient statistic, $\theta$ is a canonical parameter, and $A(...
7 votes
1 answer
1k views

Definition of exponential family with dispersion parameter

I was recently reading a discussion of generalized linear models that considered the response to come from an exponential family with a dispersion parameter so $$ f(y|\theta,\phi) = \exp\left(\frac{y\...
  • 621
7 votes
1 answer
3k views

Distribution of sum of independent exponentials with random number of summands

Let $\tau_i\sim\exp\left(\lambda\right)$ be independent and identically distributed exponentials with parameter $\lambda$. Then, for given $n$, the sum of these values $$T_n := \sum_{i=0}^n \tau_i$$ ...
  • 1,162
7 votes
1 answer
601 views

Is the negative exponential distribution a member of the exponential family?

Please correct me if I am wrong. The general form of $k$-parameter exponential family is $$f(x;\boldsymbol{\theta}) = a(\boldsymbol{\theta})g(x) \exp\{\sum_{i=1}^{k}b(\boldsymbol{\theta}) R_i(x)\}$$ ...
  • 398
7 votes
1 answer
409 views

How do I find the UMVUE of $\sqrt{\alpha}$ here?

new user here self-studying some mathematical statistics. I came across this problem and am stuck. Problem: Suppose that for $i = 1, ... , n$, the positive random variables $X_i$ are independent and ...
7 votes
2 answers
5k views

Exponential family form of multinomial distribution

I feel like this must be a duplicate, but I don't know the magic words to find the appropriate post... The multinomial distribution is a member of the exponential family. I am used to seeing the "...
  • 153
7 votes
2 answers
2k views

Intuition for why the (log) partition function matters?

I'm on a quest for the intuition behind the fact that theoretical introductions to approximate inference focus so much on the log partition function. Say we have a regular exponential family $$p(\...
  • 121
7 votes
1 answer
5k views

What is exponential family criterion to test the sufficiency and completeness of an estimator?

I am struggling to understand the following result from Casella and Berger about sufficiency and completeness for exponential families: Let $X_{1},X_{2},...,X_{n}$ be iid observations from an ...
  • 1,030

1
2 3 4 5
8