9
$\begingroup$

An exponential family distribution $p$ in the canonical form can be written as

$p(x|\theta) = h(x)\exp(\theta^\top T(x) - A(\theta))$

where $A(\theta)$ is the log partition function, $T(x)$ is the sufficient statistics, and $h(x)$ is the base measure (according to this Wikipedia page). For simplicity, let us consider a one dimensional $x$.

What is the restriction of the form of $h(x)$ ? My intuition tells me that $h(x)$ cannot be arbitrary because otherwise we can set $T(x)=0$, and leave all the "work" to $h(x)$.

For example, I understand that a Student's t is not in the exponential family. Let $t(x)$ be the density of a t distribution. If we set $T(x)=0$ and $h(x) = t(x)$, then

$A(\theta) = \log \int h(x)\exp(\theta^\top 0) \,dx = \log 1 = 0$,

and $p(x|\theta) = h(x) = t(x)$ implying that the t is in the exponential family.

What did I miss here ?

$\endgroup$
4
  • 1
    $\begingroup$ I believe the point isn't whether any given measure is part of an exponential family, for just the reasons you've noted, but whether a family is an exponential family. $\endgroup$ May 7, 2015 at 19:24
  • $\begingroup$ I do not understand. What do you mean by "measure" ? Do you mean $h(x)$ or $p(x|\theta)$ or something else ? $\endgroup$
    – wij
    May 7, 2015 at 19:45
  • $\begingroup$ I meant probability measure, but I should have said distribution, i.e. I was talking about $p(x | \theta)$. $\endgroup$ May 7, 2015 at 19:48
  • $\begingroup$ This Stanford machine learning video on youtube explains it in a very good way Video Link $\endgroup$ Jun 13, 2021 at 7:55

2 Answers 2

5
$\begingroup$

Since $p(x|\theta)$ must integrate to 1, $h(x)$ must be non-negative, but that's the only restriction (according to page 111 in this book).

However, I think the question highlights a common confusion (at least one that I've had before). There isn't just one exponential family of distributions. Rather, there are many such exponential families as mentioned in the Exponential family Wikipedia article:

exponential families are in a sense very natural sets of distributions to consider.

The choice of the functions $h$ and $T$ specify the exponential family (i.e. model) and the parameter vector $\theta$ corresponds to a particular member (i.e. distribution) in that family.

Indeed, if you choose a some fixed degrees of freedom for the t-distribution (let's say $\nu = 3$), you could as you say let $T(x) = 0$ and $h(x) = t(x|\nu=3)$ which, following the formula on from the Student's t-distribution Wikipedia article, should give $$ h(x) = \frac{1} {\sqrt{3\pi}\,\Gamma(\frac{3}{2})} \left(1+\frac{x^2}{3} \right)^{-2}\!.$$

However, this doesn't give you the family of t-distributions, i.e. the set of functions $\{t(\cdot|\nu) : \nu > 0\}$. With this base distribution, you could construct a more interesting exponential family of distributions by using more interesting sufficient statistics $T(x)$, but you would not be able to design $T$ such that parameters $\theta$ would correspond to the $\nu$ parameter of the t-distribution.

$\endgroup$
2
  • $\begingroup$ I'd also mention that, while we can "design" $T(x)$, it is defined by Fisher as "a statistic... with respect to... [$f(x; \theta$)] if no other statistic that can be calculated from the same sample provides any additional information as to the value of the parameter". Since forcing the T distribution into the exponential family form requires us to set $T(x)$ = 0, and $A(\theta)$ = 0, you're saying that no data provides information about $\theta$. In an exponential family parametrized by $\theta$, distributions from the family are unidentifiable. $\endgroup$ Oct 21, 2018 at 21:57
  • $\begingroup$ @JakeStevens-Haas: the answer explains that an (infinity of) exponential family(ies) can be built out of this choice of function $h(x)$, which essentially modifies the dominating measure from the Lebesgue measure $\text d\mu$ to the Student measure $h(x)~\text d\mu$. Each of these exponential families is associated with a specific $T(x)$ in the exponential part and none corresponds to the Student's $t$ family. $\endgroup$
    – Xi'an
    Jul 29, 2021 at 3:45
1
$\begingroup$

The central restriction is that $h(x)$ cannot depend on the parameters $\theta$. Intuitively, the base measure is the distribution of $x$ when the exponential term is given no weight i.e. $\theta=0$. As mentioned in a comment to your question, this is related to the distinction between a family of distributions being an exponential family in itself vs. any single distribution being the member of some family.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.