Questions tagged [exponential-family]

A set of distributions (eg, normal, $\chi^2$, Poisson, etc) that share a specific form. Many of the distributions in the exponential family are standard, workhorse distributions in statistics, w/ convenient statistical properties.

Filter by
Sorted by
Tagged with
6
votes
2answers
763 views

Distribution of sampling without replacement

Consider $N$ items with associated weights $w_i$. Each time, we sample one item from the remainder without replacement and the sampling probability is proportional to the weights. Continue sampling ...
3
votes
1answer
823 views

Is logistic regression a generalized linear model for more than two classes?

Suppose that $K>2$ denote the number of classes and $Y$ be the class label that follows the logistic regression model for a given $X \in \mathbb{R}^d$ as follows: $$ P(Y=k|X)= \frac{e^{\beta_k^\top ...
1
vote
1answer
230 views

use inverse Wishart for variance in MCMC

When you have a posterior that looks like as one step in Gibber Sampler $P(\xi | \Sigma_\xi, \theta) ∝ exp\{-1/2 \xi\Sigma_\xi^{-1}\xi\}P(data | \xi, \theta)$ Do you always assume inverse Wishart ...
8
votes
1answer
3k views

Why do we assume the exponential family in the GLM context?

When I first learned about Generalized Linear Models I thought that the assumption that the dependent variable follows some distribution from the exponential family was made to simplify calculations. ...
0
votes
1answer
101 views

how the conditional distribution of $(\sum y_i^2 |\sum y_i, )$ form an exponential family

Suppose a sample from the normal distribution, the canonical statistic is: $\boldsymbol{t(y)} = (\sum y_i, \sum y_i^2) = (v,u)$. The distribution $f(u|v;\theta)$ forms an exponential family. In ...
0
votes
1answer
2k views

Cramer-Rao lower bound in a Gamma distribution

I have this Gamma density function: $$\frac{1}{6\theta^8} x^3 \exp\Big[\frac{-1}{\theta^2}x\Big]$$ If I calculate the MSE I have that: $$\hat\theta=\frac{1}{2}\sqrt{\frac{1}{n}\sum{x_i}}$$ Now, if ...
2
votes
0answers
749 views

Minimal sufficient statistic for two normal distributions

Let $X_1, . . . , X_m, Y_1, . . . , Y_n$ be independent with $X_i ∼ N(ξ, σ^2)$ and $ Y_j ∼ N(η, τ^2).$ What is the minimal sufficient statistic for $(ξ,η,σ^2)$ where $σ^2 = τ^2$? I've seen MSS ...
4
votes
1answer
223 views

Distribution for the canonical statistic of exponential family

A statistical model for the data set $\bf{y}$ is an exponential family with canonical parameter vector $\theta = (\theta_1,.. \theta_k)$ and canonical statistic $\bf{t(y)} $=$(t_1(\boldsymbol{y}),.....
5
votes
2answers
834 views

Exercises in EM with censored data

I'm in the process of working through Exercises in EM. I am having trouble understanding the first exercise. In particular, I am struggling with deriving the conditional expected value (2) and (3). ...
3
votes
1answer
5k views

linear regression on exponential distributed dependent variable

suppose that I want to use linear regression on a data where independent variables x1,x2,...xn are all more or less normally distributed, while the dependent variable y is almost exponentially ...
3
votes
3answers
172 views

Distribution for which the log-likelihood evaluated at the ML estimate is not equal to the expected log-likelihood evaluated at the ML estimate

Suppose that for $t=1,\dots,T$, $x_t$ is an i.i.d. draw from a continuous distribution with p.d.f. $f(x_t;\theta_0)$. Let $l(\theta;X):=\sum_{t=1}^T{\log{f(x_t;\theta)}}$ be the corresponding log-...
3
votes
2answers
4k views

Is Weibull distribution a exponential family?

I'm wondering is Weibull distribution a exponential family?
2
votes
1answer
1k views

does logistic distribution belongs to exponential family

Let $X$ have the logistic distribution with the PDF $$f(x) = \frac{\exp(-x-θ)}{(1+\exp(-x-θ))^{2}}$$ Does $f(x)$ belong to the exponential family? My solution is $\exp[(-2)\cdot \ln(1+\exp\{-x-θ\})-x-...
0
votes
0answers
547 views

example of uniformly most powerful test

let X be a single observation from the density $f(x;\theta)$ =$ \theta x^{\theta -1} I_{(0,1)}(x)$ is there a UMP size-$\alpha$ test for testing $H_0 :\theta \ge \frac{1}{2} $ V/S $ H_1 : \theta <...
0
votes
0answers
38 views

How to approach expectation of order statistic conditioned on known minimum?

Say you have an $n \times n$ matrix whose entries follow a standard exponential distribution, $A_{i,j} \sim \text{Exp}(1)$ for $1 \le i,j \le n$. You find the minimum value, $x_{i:n^2}$, and remove ...
7
votes
1answer
435 views

Is the expectation of the sufficient statistics $S(X)$ transverse the whole space in an exponential family?

An exponential family is defined using two ingredients: - a base density $q_0(x)$ - a number of sufficient statistics $S_i(x)$ The family is all probability densities which can be written as: $$ q(x| ...
1
vote
0answers
70 views

Fitting a GLM – whether to estimate mean or dispersion first

I'm trying to understand how fitting GLMs works. I asked before on math.SE but this community is probably more appropriate. My confusion is the following: The MLE for $\hat\mu$ uses $\phi$. My books ...
5
votes
1answer
995 views

KL Divergence, Bregman, and uniqueness

While reading the following paper on Bregman Divergence (link) Banerjee, Arindam, et al. "Clustering with Bregman divergences." Journal of machine learning research 6.Oct (2005): 1705-1749. In ...
3
votes
2answers
793 views

How can variance be non constant for Bernoulli and Poisson

Until now I had learned that the variances of Bernoulli and Poisson random variables are $p(1-p)$ and $λ$ and that for fixed $p$ and $λ$, these variances are constant. Now, introducing glm, my course ...
5
votes
1answer
939 views

What is sufficient statistics in the context of generalized linear models

the three assumptions Andrew Ng makes when deriving GLM are: $y\mid x;\theta \sim \operatorname{ExponentialFamily}(\eta)$ Our goal is to predict the expected value of $T(y)$ given $x$. Since in most ...
1
vote
0answers
288 views

Gaussian Exponential Sum Approximation

Let $x_j,\ j\in\{1,\cdots,n\}$ be independent standard Gaussian random variables and $a_j,\ j\in\{1,\cdots,n\}$ be constants. What would a function $f$ and a standard Gaussian random variable $x_0$ ...
3
votes
1answer
272 views

Formula for the covariance of logs in exponential families

I could not understand and don't see how the author derived equation (16). The Covariance is with respect to a pdf $\mu_k$ for the random vector $u$. The variable $\xi$ is a ratio of two densities and ...
1
vote
1answer
47 views

Renewal process with hypoexponentially distributed holding times

In order to test my program I need to calculate 95% confidence intervals (preferably even CDF) for a distribution of a renewal process with hypoexponentially distributed holding times: $X = X_{\...
2
votes
0answers
205 views

Sufficient statistics in multiparameter exponential family

I'm trying to work through a theorem in the Lehmann statistical inference book and I'm confused about a proof. They are proving that a set of tests are UMP unbiased level-alpha tests for a series of ...
0
votes
1answer
938 views

How should you express a negative binomial distribution (\w gamma function) in an exponential family form?

How should you express a negative binomial distribution (\w gamma function), i.e. $$f(y_i, \mu, \phi) = \frac{\Gamma (y+ \phi)}{\Gamma(\phi) \Gamma(y+1)}(\frac{\mu}{\mu + \phi})^y (\frac{\phi}{\mu + \...
3
votes
1answer
146 views

Is there a good exponential-family PDF with analytic CDF for modeling complicated, noisy backgrounds?

Through simulation I'm comparing several methods for isolating a signal distribution from complicated, noisy background (two components in a mixture). What I need is a good PDF model for this noisy ...
6
votes
1answer
210 views

Exotic distribution

During the development of a variational mean field algorithm I have found a distribution with the form: $q(x) \propto x e^{-ax^2 +bx}$ with $x\in(0,+\infty)$ Does such a distribution have a name? ...
10
votes
1answer
5k views

Why is the continuous uniform distribution not an exponential family?

In class I've been given that a family of distributions $\{P_{\theta} : \theta \in \Theta\}, \Theta \in \mathbb{R}^{k} $ is an exponential family if there exits real-valued functions $\eta_{1}, ... , \...
1
vote
0answers
121 views

Exponential Family

I want to check if the PDF belongs to the exponential family: $$f(x|v) = C \exp-\sum\limits_{i=1}^n(x_{i}-v)^4$$ How can this fit into this formula (to check if the pdf belongs to the exponential ...
7
votes
1answer
401 views

Generalized Linear Models - What's special about the exponential family?

In Generalized Linear Models the conditional distribution of the response variable has to belong to the exponential family. Why is this restriction important? What property would a regression model ...
0
votes
0answers
3k views

is Pareto distribution exponential dispersion family and the form is unique?

I am trying to show that Pareto distribution with $$f(y;\alpha)=\alpha y^{-\alpha-1} $$ is exponential dispersion (ED) family which means that it can be rewritten as: $$f(y;\theta,\phi)={\rm exp}\{\...
0
votes
1answer
1k views

Calculating the parameters of a Normal distribution using alpha and beta from Inverse-gamma (conjugate prior)

How is it possible to calculate the variance $\sigma^2$ for the Normal distribution if only $\alpha$ and $\beta$ (based on data) from the Inverse-gamma distribution are available? I followed the ...
1
vote
0answers
201 views

Running an exponential random graph model (ERGM) with lagged covariates

at the moment I am trying to fit an exponential random graph model (ERGM) in R with the function 'ergm' that comes with the 'statnet' package. Here I have two questions. One is of technical nature and ...
3
votes
1answer
206 views

What do percentiles tell us about exponential families

I have a question in my lecture slides which asks the following Exponential families of distributions, such as the normal, Poisson, and binomial distributions, are characterized by a very fast (...
2
votes
1answer
6k views

Sufficient statistic for bivariate or multivariate normal

Here on page 7, example 2.7. The claim is that sufficient statistics for $d$ dimensional multivariate normal $\mathbf{x}_i \sim N(\vec{\mu}, \Sigma)$ is $$\left(n^{-1}\sum_{i=1}^n \mathbf{x}_i, \hat{\...
4
votes
1answer
108 views

In GLM do we try to model E(T(y)) or E(y)?

I'm trying to follow Andrew NG cs course on supervised learning. He defines the exponential family as: $$ p(y;\eta) = b(y)exp(\eta T(y) -a(\eta)) $$ and then continues to say that "our goal is to ...
4
votes
1answer
277 views

Can Cook's distance plot only be used for least squares regression?

If Cook's distance can only be used for least squares regression, what are some alternatives that will give me a similar plot for a Gamma model or any regression model from the exponential family?
2
votes
1answer
546 views

Reasoning behind reciprocal link function for exponential regression

The reciprocal link function in exponential regression doesn't constrain to positive values, whilst on the other hand it does thoroughly space out points close to zero when fitting the linear ...
3
votes
1answer
729 views

GLM (conditional/unconditional) distribution

Based on my readings about GLM, I am pretty sure that when we say the distribution of the response variable $y$ is a member of exponential family of distribution, what we really mean is that ...
3
votes
2answers
263 views

Why computing P(x,D) is simpler than P(x|D) in exponential bayesian networks?

I am reading this tutorial on variational inference and wonder why the statement in the question title which is mentioned on page 3 is true.
4
votes
1answer
603 views

Can a generalized linear model use shifted exponential as residual distribution?

I am facing a modeling problem: $t_{ij} = D_i + T_j + \epsilon_{ij}, i=0...641, j\in\mathbb{N}$ where $\epsilon_{ij}$ follows exponential distribution, $\epsilon_{ij} \sim \lambda e^{-\lambda t}, \...
3
votes
1answer
788 views

What r parameter is used in a negative binomial regression?

From my understanding of the negative binomial regression, we have $Y_i|X_i; \theta$ is distributed $Neg Binom (r_i, p_i)$, where $r_i$ is known and fixed (analogous to a fixed $\sigma^2$ when we ...
6
votes
2answers
2k views

Help Deriving Variance Function - Binomial GLM

I'm having difficulty replicating/deriving a result in GLM's for Binomial data. That is, if $Y \sim Bin(n, \mu)$ and we put the distribution of $Y/n$ into exponential family form (with a dispersion ...
4
votes
1answer
478 views

Are there any non exponential family distributions with conjugate priors? [duplicate]

I believed I had been taught that only exponential family distributions have conjugate priors but I have recently read that ' all exponential family distributions have conjugate priors', leaving the ...
0
votes
1answer
335 views

Show that Weibull distribution belongs to a one dimensional exponential family

It is given that $f_\eta(y) = h(y)exp(\eta T(y)-A^*(\eta))$ $P_Y(y)= \frac{k}{\lambda} (\frac{y}{\lambda})^{k-1}exp(-(\frac{y}{\lambda})^k)$ What i did was by arranging $P_Y(y)$ to get $\frac{k}{\...
2
votes
0answers
75 views

Expected time between two events

I'm having trouble with the following problem: Consider a game between two players A and B. Player A must complete three tasks each of which take an exponentially distributed amount of time with ...
1
vote
0answers
120 views

Computing the partition-function of an exponential family member

I am working on a Monte Carlo Expectation Propagation problem. In that context I got the following property: $ I = \sum\limits_i w^{(i)} \log p_\eta(x^{(i)}) $ where $\{w^{(i)}\}_i$ are weights, $...
1
vote
1answer
880 views

Zero-truncated Poisson model

In the theory of generalised linear models, you may use the exponential family to find the mean and variance of certain distributions. How would the mean and expectation of the zero-truncated Poisson ...
-1
votes
1answer
122 views

Does the following pmf belong to the exponential family?

I recently saw* a pmf: $f(y)=\frac{\mu^y}{(y!)^\theta z(\mu,\theta)}$, where $z(\mu,\theta) = \sum_{i=0}^{\infty}\frac{\mu^i}{(i!)^\theta}$. * It is a bonus question on a homework assignment. My ...
2
votes
0answers
149 views

Variance of Distributions from the Exponential Family

I want to understand how the variance of an exponential family behaves. To take a very concrete example. Let consider the unit ball $B$ in d dimensions. Consider the following distribution over unit ...

1
3 4
5
6 7