# Exponential family form of multinomial distribution

I feel like this must be a duplicate, but I don't know the magic words to find the appropriate post...

The multinomial distribution is a member of the exponential family. I am used to seeing the "standard" formulation of the multinomial distribution as:

$$p(x|\pi) = \frac{n!}{\prod_{i=1}^m x_i!}\prod_{i=1}^m \pi_i^{x_i}$$

I have seen multiple sources express the multinomial, in the form of an exponential family member as:

$$p(x|\pi) = \exp \{\sum_{i=1}^m x_i \ln \pi_i \}$$

My question is, what happened to the multinomial coefficient $\frac{n!}{\prod_{i=1}^m x_i!}$ ??? If I try to go from the exponential family formulation back to the standard form, I get:

$$p(x|\pi) = \prod_{i=1}^m \pi_i^{x_i}$$

From Kevin Murphy's book, it appears that this exponential formulation only holds for a single trial, in which case the multinomial coefficient is 1.

• I corrected the term $\sum_{i=1}^m x_i!$ into $\prod_{i=1}^m x_i!$ – Xi'an Oct 15 '15 at 7:52

Exponential families are characterised by their densities, which are such that the interaction between the outcome $x$ of the random variable and the parameter $\theta$ occurs in an exponential scalar product, $$\exp\{T(\theta)^\text{T} S(x)\}$$ The other terms in the density are the normalising constant $C(\theta)=\exp\{-\psi(\theta)\}$ and a function of $x$, $h(x)$, which comes to complement the dominating measure $\text{d}\nu(x)$. But all that matters is the product $h(x)\text{d}\nu(x)$ which can be seen as a new measure $\text{d}\nu´(x)$.

So in the multinomial example, the term $${n \choose x_1 \cdots x_m}=\frac{n!}{\prod_{i=1}^m x_i!}$$ can either be seen as $h(x_1,\ldots,x_m)$ completing the counting measure [which qualifies as "standard" in your terms] on $$\left\{(x_1,\ldots,x_m)\in\mathbb{N}^m;\sum_{i=1}^m x_i=n\right\}$$ or as part of a new measure on that set.

• So in other words, it is still there, but is often "ignored" by books and literature as they omit the $h()$ term ? – Xavier Bourret Sicotte Jun 1 '18 at 10:05
• hmm, but is still doesn't explain why the coefficient n!∏mi=1xi! is omitted. @Xavier Bourret Sicotte have u found a satisfied explanation? – lnshi Dec 8 '18 at 2:32

I was trying to build the softmax regression from scratch, and got stuck here also: why all the posts just silently omit the coefficient $$\frac{n!}{x_1! x_2! \dots x_k!}$$ totally, after some research and thinking i share my understanding here:

In reality, we always use a special form of the multinomial distribution with $$k>2$$ and $$n=1$$, that is categorical distribution, and since $$n=1$$, then the coefficient $$\frac{n!}{x_1! x_2! \dots x_k!}$$ of the multinomial distribution's PMF will be always all 1, that is why we can omit it, more details:

$$\begin{cases} \text{class 1 is chosen: } \frac{1!}{1! \cdot 0! \cdot 0! \dots 0!} = 1 \\ \text{class 2 is chosen: } \frac{1!}{0! \cdot 1! \cdot 0! \dots 0!} = 1 \\ \text{class 3 is chosen: } \frac{1!}{0! \cdot 0! \cdot 1! \dots 0!} = 1 \\ \vdots \end{cases}$$

For getting more details and a better full pic of understanding u can refer to my post GLM and exponential family distributions -> Why the PMF has no coefficient

Especially pay attention to understand what does the $$x_i$$ stand for in multinomial distribution from this section

• Would you be better asking this as a new question since an answer cannot get an answer? – mdewey Dec 8 '18 at 14:47
• @mdewey It reads like an effort to answer the question. It appears only to address a very special case of the question, though, so whether it is correct or useful may be for each reader to determine. – whuber Dec 8 '18 at 15:13
• @mdewey i was trying to answer the question, not seeking any answer, since the accepted answer totally doesn't address the main concern of the question: why the coefficient is silently omitted, i just share my understanding to may help someone – lnshi Dec 9 '18 at 2:10