Questions tagged [exponential-family]

A set of distributions (eg, normal, $\chi^2$, Poisson, etc) that share a specific form. Many of the distributions in the exponential family are standard, workhorse distributions in statistics, w/ convenient statistical properties.

Filter by
Sorted by
Tagged with
29
votes
1answer
2k views

When if ever is a median statistic a sufficient statistic?

I came across a casual remark on The Chemical Statistician that a sample median could often be a choice for a sufficient statistic but, besides the obvious case of one or two observations where it ...
21
votes
3answers
3k views

Why doesn't the exponential family include all distributions?

I am reading the book: Bishop, Pattern Recognition and Machine Learning (2006) which defines the exponential family as distributions of the form (Eq. 2.194): $$ p(\mathbf x|\boldsymbol \eta) = h(\...
21
votes
2answers
2k views

Advantages of the Exponential Family: why should we study it and use it?

So here I am studying inference. I would like that someone could enumerate the advantages of the exponential family. By exponential family, I mean the distributions which are given as \begin{align*} f(...
19
votes
2answers
4k views

Does log likelihood in GLM have guaranteed convergence to global maxima?

My questions are: Are generalized linear models (GLMs) guaranteed to converge to a global maximum? If so, why? Furthermore, what constraints are there on the link function to insure convexity? My ...
17
votes
0answers
761 views

Is there a general expression for ancillary statistics in exponential families?

An i.i.d sample $X_1,\dots,X_n$ from a scale family with c.d.f. $F(\frac{x}{\sigma})$ has $S(X)$ as an ancillary statistic if $S(X)$ depends on the sample only through $\frac{X_1}{X_n},\cdots,\frac{X_{...
16
votes
2answers
6k views

Kullback–Leibler divergence between two gamma distributions

Choosing to parameterize the gamma distribution $\Gamma(b,c)$ by the pdf $g(x;b,c) = \frac{1}{\Gamma(c)}\frac{x^{c-1}}{b^c}e^{-x/b}$ The Kullback-Leibler divergence between $\Gamma(b_q,c_q)$ and $\...
16
votes
3answers
10k views

Poisson is to exponential as Gamma-Poisson is to what?

A Poisson distribution can measure events per unit time, and the parameter is $\lambda$. The exponential distribution measures the time until next event, with the parameter $\frac{1}{\lambda}$. One ...
15
votes
3answers
4k views

Definition of family of a distribution?

Does a family of a distribution have a different definition for statistics than in other disciplines? In general, a family of curves is a set of curves, each of which is given by a function or ...
15
votes
2answers
268 views

Derivation of normalizing transform for GLMs

$\newcommand{\E}{\mathbb{E}}$How is the $A(\cdot) = \displaystyle\int\frac{du}{V^{1/3}(\mu)}$ normalizing transform for the exponential family derived? More specifically: I tried to follow the ...
13
votes
4answers
2k views

What is the rationale behind the exponential family of distributions?

From elementary probability course, the probability distributions such as Gaussian, Poisson or exponential all have a good motivation. After staring at the formula of the exponential family ...
13
votes
1answer
744 views

Aside from the exponential family, where else can conjugate priors come from?

Do all conjugate priors have to come from the exponential family? If not, what other families are known to have/produce conjugate priors?
12
votes
1answer
1k views

Do the mean and the variance always exist for exponential family distributions?

Assume a scalar random variable $X$ belongs to a vector-parameter exponential family with p.d.f. $$ f_X(x|\boldsymbol \theta) = h(x) \exp\left(\sum_{i=1}^s \eta_i({\boldsymbol \theta}) T_i(x) - A({\...
12
votes
1answer
887 views

Does a canonical link function always exist for a Generalized Linear Model (GLM)?

In GLM, assuming a scalar $Y$ and $\theta$ for the underlying distribution with p.d.f. $$f_Y(y | \theta, \tau) = h(y,\tau) \exp{\left(\frac{\theta y - A(\theta)}{d(\tau)} \right)}$$ It can be shown ...
11
votes
1answer
3k views

How to derive the conjugate prior of an exponential family distribution

I am trying to derive the conjugate prior of the univariate Gaussian distribution over both the mean and the precision. I know that the prior I'm looking for is the normal-gamma distribution, but the ...
11
votes
1answer
801 views

Gaussian Like distribution with higher order moments

For the Gaussian distribution with unknown mean and variance, the sufficient statistics in the standard exponential family form is $T(x)=(x,x^2)$. I have a distribution that has $T(x)=(x,x^2,...,x^{2N}...
10
votes
1answer
8k views

ML estimate of exponential distribution (with censored data)

In Survival Analysis, you assume the survival time of a r.v. $X_i$ to be exponentially distributed. Considering now that I have $x_1,\dots,x_n$ "outcomes" of i.i.d r.v.'s $X_i$. Only some proportion ...
10
votes
1answer
5k views

Why is the continuous uniform distribution not an exponential family?

In class I've been given that a family of distributions $\{P_{\theta} : \theta \in \Theta\}, \Theta \in \mathbb{R}^{k} $ is an exponential family if there exits real-valued functions $\eta_{1}, ... , \...
10
votes
1answer
3k views

Exponential Family: Observed vs. Expected Sufficient Statistics

My question arises from reading reading Minka's "Estimating a Dirichlet Distribution", which states the following without proof in the context of deriving a maximum-likelihood estimator for a ...
10
votes
1answer
991 views

Find UMVUE of $\frac{1}{\theta}$ where $f_X(x\mid\theta) =\theta(1 +x)^{−(1+\theta)}I_{(0,\infty)}(x)$

Let $X_1, X_2, . . . , X_n$ be iid random variables having pdf $$f_X(x\mid\theta) =\theta(1 +x)^{−(1+\theta)}I_{(0,\infty)}(x)$$ where $\theta >0$. Give the UMVUE of $\frac{1}{\theta}$ ...
10
votes
1answer
842 views

Unbiased estimator with minimum variance for $1/\theta$

Let$ X_1, ...,X_n$ be a random sample feom a distribution $Geometric(\theta)$ for $0<\theta<1$. I.e, $$p_{\theta}(x)=\theta(1-\theta)^{x-1} I_{\{1,2,...\}}(x)$$ Find the unbiased estimator ...
9
votes
1answer
1k views

What is meant by the term 'exponential family'? Why it is named so?

I have come across the term exponential family. The Bernoulli, Gaussian and many more distributions come under this exponential family. What would be the commonalities between them?
9
votes
2answers
2k views

Why is the mixtures of conjugate priors important?

I have questions about the mixture of conjugate priors. I learned and saw the mixture of conjugate priors a couple of times when I am learning bayesian. I am wondering why this theorem is such ...
9
votes
2answers
710 views

Find the joint distribution of $X_1$ and $\sum_{i=1}^n X_i$

This question is from Robert Hogg's Introduction to Mathematical Statistics 6th version question 7.6.7. The problem is : Let a random sample of size $n$ be taken from a distribution with the pdf $...
9
votes
2answers
893 views

A stochastically increasing exponential family for which $\lim_{\theta\rightarrow\inf\Theta}\mbox{P}_\theta(X\leq x)\neq 1$

Question A little something that I've been wondering about for a while: Let $P_\theta$ be a stochastically increasing (one-parameter) exponential family on the sample space $\mathcal{X}$ with $\...
9
votes
1answer
2k views

Is the negative binomial not expressible as in the exponential family if there are 2 unknowns?

I had a homework assignment to express the negative binomial distribution as an exponential family of distributions given that the dispersion parameter was a known constant. This was fairly easy, but ...
8
votes
1answer
15k views

How should you express a negative binomial distribution in an exponential family form?

When $X$ is $X_1$,...$X_n$, how do you express the following negative binomial distribution in an exponential family form? $$ f(k;r,p)\equiv\text{Pr}(X=k)=\binom{k+r-1}{k}(1-p)^{r}p^k~~~~~\text{for}~...
8
votes
1answer
279 views

Does a sufficient statistic imply the existence of a conjugate prior?

In the comments on this answer, user Scortchi asks: So iff there's a sufficient statistic of constant dimension, there's a conjugate prior? As far as I know this didn't get a complete answer, so I'm ...
8
votes
2answers
1k views

Checking whether a density is exponential family

Trying to prove that this doesn't belong to exponential family. $f(y|a)=4\frac{(y+a)}{(1+4a)} ; 0 < y < 1 , a>0$ Here is my approach: $$f(y|a) = 4(y+a)e^{-log(1+4a)}$$ $$f(y|a) = (4y)(1+\...
8
votes
2answers
4k views

Finding UMVUE of $\theta e^{-\theta}$ where $X_i\sim\text{Pois}(\theta)$

Suppose $X_1, X_2, . . . , X_n$ are i.i.d Poisson ($\theta$) random variables, where $\theta\in(0,\infty)$. Give the UMVUE of $\theta e^{-\theta}$ I found a similar problem here. I have that the ...
8
votes
1answer
3k views

Why do we assume the exponential family in the GLM context?

When I first learned about Generalized Linear Models I thought that the assumption that the dependent variable follows some distribution from the exponential family was made to simplify calculations. ...
8
votes
1answer
204 views

Support vector machines (SVMs) are the zero temperature limit of logistic regression?

I was had a quick discussion recently with a knowledgeable friend who mentioned that SVMs are the zero temperature limit of logistic regression. The rationale involved marginal polytopes and fenchel ...
7
votes
3answers
2k views

Which parameter should be considered as “scale” parameter for Gamma distribution?

From Wikipedia and probably all statistics textbooks, we know that in the density of a Gamma random variable $$f(x; k, \theta) = \frac{1}{\Gamma(k)\theta^k}x^{k - 1}e^{-\frac{x}{\theta}}, \quad x > ...
7
votes
1answer
176 views

Is $\mathrm{binomial}(n, p)$ family both full and curved for $n$ fixed?

Let $n$ be a fixed positive integer. The binomial$(n, p)$ family is given by $$f(x|p)=\tbinom{n}{x}p^x(1-p)^{n-x}\tag{1}.$$ We may rewrite (1) as $$f(x|p)=\tbinom{n}{x}(1-p)^n\exp\left[x\log\frac{p}{1-...
7
votes
2answers
746 views

Sufficient Statistic for non-exponential family distribution

Question: Let $X_1,X_2,\ldots,X_n$ be an iid sample from $N(\theta , 4 \theta^2 )$. I want to show that this model is not a member of the exponential family and to find a sufficient statistic for $\...
7
votes
1answer
3k views

Distribution of sum of independent exponentials with random number of summands

Let $\tau_i\sim\exp\left(\lambda\right)$ be independent and identically distributed exponentials with parameter $\lambda$. Then, for given $n$, the sum of these values $$T_n := \sum_{i=0}^n \tau_i$$ ...
7
votes
1answer
391 views

Is the negative exponential distribution a member of the exponential family?

Please correct me if I am wrong. The general form of $k$-parameter exponential family is $$f(x;\boldsymbol{\theta}) = a(\boldsymbol{\theta})g(x) \exp\{\sum_{i=1}^{k}b(\boldsymbol{\theta}) R_i(x)\}$$ ...
7
votes
1answer
2k views

Conditional distribution for Exponential family

We have a random variable $X$ that belongs to the exponential family with p.d.f. $$ P_X(x|\boldsymbol \theta) = h(x) \exp\left(\eta({\boldsymbol \theta}) . T(x) - A({\boldsymbol \theta}) \right) $$ ...
7
votes
1answer
222 views

How do I find the UMVUE of $\sqrt{\alpha}$ here?

new user here self-studying some mathematical statistics. I came across this problem and am stuck. Problem: Suppose that for $i = 1, ... , n$, the positive random variables $X_i$ are independent and ...
7
votes
1answer
4k views

What is exponential family criterion to test the sufficiency and completeness of an estimator?

I am struggling to understand the following result from Casella and Berger about sufficiency and completeness for exponential families: Let $X_{1},X_{2},...,X_{n}$ be iid observations from an ...
7
votes
1answer
434 views

Is the expectation of the sufficient statistics $S(X)$ transverse the whole space in an exponential family?

An exponential family is defined using two ingredients: - a base density $q_0(x)$ - a number of sufficient statistics $S_i(x)$ The family is all probability densities which can be written as: $$ q(x| ...
7
votes
2answers
4k views

Exponential family form of multinomial distribution

I feel like this must be a duplicate, but I don't know the magic words to find the appropriate post... The multinomial distribution is a member of the exponential family. I am used to seeing the "...
7
votes
2answers
2k views

Role of base measure in exponential family

An exponential family distribution $p$ in the canonical form can be written as $p(x|\theta) = h(x)\exp(\theta^\top T(x) - A(\theta))$ where $A(\theta)$ is the log partition function, $T(x)$ is the ...
7
votes
2answers
475 views

Intuition for why the (log) partition function matters?

I'm on a quest for the intuition behind the fact that theoretical introductions to approximate inference focus so much on the log partition function. Say we have a regular exponential family $$p(\...
7
votes
1answer
224 views

The distribution of a sufficient statistic

If I understand correctly, a distribution in the exponential family... $$\underline X\sim f_{\underline\theta}(\underline x) = exp\{\sum\limits_{i}\eta_i(\underline\theta)T_i(\underline x)-B(\...
7
votes
1answer
182 views

Closed form function relating $\mu$ to the natural parameter for the logarithmic series distribution?

While answering another question here, I mentioned the logarithmic series distribution as a possible model for species per genus. In the course of looking at the pmf while answering that I realized ...
7
votes
1answer
400 views

Generalized Linear Models - What's special about the exponential family?

In Generalized Linear Models the conditional distribution of the response variable has to belong to the exponential family. Why is this restriction important? What property would a regression model ...
7
votes
2answers
1k views

Are log-linear models exponential models?

What is usually referred to as "log-linear models"? Is a log-linear model an exponential model where the normalization constant is 1 (since its logarithm needs to be a linear function)? Or is there ...
6
votes
1answer
169 views

Name and interpretation of “$h(x)$” in exponential family

The exponential family is defined (in many sources) as: $$p(x | \theta) = h(x) \exp\{\theta^TT(x) - A(\theta)\}$$ where: $T(x)$ is a sufficient statistic, $\theta$ is a canonical parameter, and $A(...
6
votes
1answer
2k views

Deriving the canonical link for a binomial distribution

I define an exponential dispersion family as any distribution whose PMF/PDF is $$f(y \mid \boldsymbol\theta) = \exp\left\{\phi[y\theta - b(\theta)] + c(y, \phi) \right\}\text{, } y \in \Omega$$ where ...
6
votes
1answer
210 views

Exotic distribution

During the development of a variational mean field algorithm I have found a distribution with the form: $q(x) \propto x e^{-ax^2 +bx}$ with $x\in(0,+\infty)$ Does such a distribution have a name? ...

1
2 3 4 5
7