1
$\begingroup$

I am going through Michael Jordan's notes on exponential families and an exponential family is defined w.r.t. functions $h(\cdot), T(\cdot)$, and parameter $\eta$ such that $$ p(x | \eta) = h(x) \exp\bigl\{\eta^\top T(x) - A(\eta)\bigr\} $$ with $$ A(\eta) = \log\int h(x) \exp\left(\eta^\top T(x) \right) \, dx. $$ The thing that strikes me as odd about this definition is that it appears too flexible due to the choice of "$h$". For instance, say we define our family in terms of $h_1, \eta_1$, and $T_1$. We could then reparameterize our family in terms of $h_2, \eta_2$, and $T_2$ such that

  1. $\eta_2 = 0$
  2. or alternatively, $T_2(x) = 0$
  3. $h_2(x) = h_1(x) \exp\left(\eta_1^\top T_1(x)\right)$

so the choices of $\eta$ and $T(\cdot)$ are not important (at least in terms of defining the density).

Moreover, it seems we might use this trick to mimic any density (not necessarily ones commonly thought of as "exponential families"), simply by letting $\eta = 0$ and making a "sneaky" choice for $h(\cdot)$.

What am I missing here?

Improve this question
$\endgroup$
5
  • 2
    $\begingroup$ Suppose I want to consider the family of distributions which are uniform on $[0,\eta]$. So $p(x\mid \eta) = \frac1\eta \mathbb I_{0 \le x \le \eta}$. Can you find suitable $h(x),T(x),A(\eta)$ to show this is an exponential family? $\endgroup$
    – Henry
    Commented Oct 22, 2019 at 23:10
  • $\begingroup$ @Henry - is the domain in your example the interval $[0,\eta]$ or $[0,+\infty)$? $\endgroup$
    – ted
    Commented Oct 23, 2019 at 0:43
  • $\begingroup$ @Henry - Assuming the domain is $\{ x : x \ge 0\}$, then for $x \in [0,\eta]$ we would let $h_2(x) = \frac{1}{\eta}$ and for $x > \eta$ we would let $h_2(x) = 0$. This should work since the denominator is $\int_0^{\infty} \frac{dx}{\eta} = \int_0^{\eta} \frac{dx}{\eta} = 1$ and the numerator is $h_2(x) = \frac{1}{\eta}$ whenever $x \in [0,\eta]$ (and is zero otherwise). $\endgroup$
    – ted
    Commented Oct 23, 2019 at 0:54
  • 3
    $\begingroup$ Your definition of $h_2(x)$ seems to depend on the value of $\eta$, and that is precisely what is not allowed for exponential families. Any dependence on the parameter $\eta$ should be in the $\exp(\cdots)$ term. Since that is always positive but here you need it to sometimes to be $0$, you can conclude that this family of uniform distributions is not an exponential family. $\endgroup$
    – Henry
    Commented Oct 23, 2019 at 7:52
  • $\begingroup$ @Henry - thanks for helping me see the light! I didn't realize that $h(\cdot)$ couldn't implicitly depend on $\eta$. $\endgroup$
    – ted
    Commented Oct 24, 2019 at 4:20

1 Answer 1

Reset to default
7
$\begingroup$

No, the definition is not vacuous. One consequence of the definition is that the support of all the distributions in the family is the same, which excludes the uniform distribution family $\mathcal{Unif}(0,\theta)$. This is easy to see, the exponential factor is never zero, and the $h$ function is the same for all the family members (your $h_2$ example in the post breaks this assumption.)

We can use exponential tilting to generate an exponential family from a random variable (if that random variable admits a moment generating function (mgf), but if we start with a standard uniform random variable, we do not get a family of uniform distributions. What we get is the family $$ f(x\,; \theta)\, = \exp(\theta x -k(\theta))\cdot \mathbb{I}(x\in (0,1)\,) $$ where $k(\theta)=\log( (e^\theta -1)/\theta )$ (and zero if $\theta=0$) is the cgf (cumulant generating function) of the standard uniform. The cgf is the logarithm of the mgf. A few members of this family is shown below:

tilted family generated by standard uniform

The code for the plot is below:

k <- function(theta) ifelse(abs(theta)<0.000001, 0.0, 
                            log( (exp(theta)-1) / theta ) )

f_tilted <- function(x, theta) ifelse( (x<1)&(x>0), 
                     exp(theta*x-k(theta)), 0)

library(RColorBrewer)

palette <- brewer.pal(9, "Greens")
theta <- seq(-2.0, 2, length.out=9)
plot(x=c(0, 1),  y=c(0, 2.5), xlab="x", ylab="tilted density", type="n")
for (ind  in 1:9 )  {
    plot( function(x)f_tilted(x, theta[ind]), 
          from=0.00001, to=0.9999, add=TRUE, col=palette[ind])
}
title("Exponential tilted family\ngenerated by standard uniform")
legend("top", legend=theta, col=palette, lwd=1)
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.