I feel like this must be a duplicate, but I don't know the magic words to find the appropriate post...

The multinomial distribution is a member of the exponential family. I am used to seeing the "standard" formulation of the multinomial distribution as:

$$p(x|\pi) = \frac{n!}{\prod_{i=1}^m x_i!}\prod_{i=1}^m \pi_i^{x_i}$$

I have seen multiple sources express the multinomial, in the form of an exponential family member as:

$$p(x|\pi) = \exp \{\sum_{i=1}^m x_i \ln \pi_i \}$$

My question is, what happened to the multinomial coefficient $\frac{n!}{\prod_{i=1}^m x_i!}$ ??? If I try to go from the exponential family formulation back to the standard form, I get:

$$p(x|\pi) = \prod_{i=1}^m \pi_i^{x_i}$$

From Kevin Murphy's book, it appears that this exponential formulation only holds for a single trial, in which case the multinomial coefficient is 1.

  • 1
    $\begingroup$ I corrected the term $\sum_{i=1}^m x_i!$ into $\prod_{i=1}^m x_i!$ $\endgroup$ – Xi'an Oct 15 '15 at 7:52

Exponential families are characterised by their densities, which are such that the interaction between the outcome $x$ of the random variable and the parameter $\theta$ occurs in an exponential scalar product, $$\exp\{T(\theta)^\text{T} S(x)\}$$ The other terms in the density are the normalising constant $C(\theta)=\exp\{-\psi(\theta)\}$ and a function of $x$, $h(x)$, which comes to complement the dominating measure $\text{d}\nu(x)$. But all that matters is the product $h(x)\text{d}\nu(x)$ which can be seen as a new measure $\text{d}\nu´(x)$.

So in the multinomial example, the term $${n \choose x_1 \cdots x_m}=\frac{n!}{\prod_{i=1}^m x_i!}$$ can either be seen as $h(x_1,\ldots,x_m)$ completing the counting measure [which qualifies as "standard" in your terms] on $$\left\{(x_1,\ldots,x_m)\in\mathbb{N}^m;\sum_{i=1}^m x_i=n\right\}$$ or as part of a new measure on that set.

| cite | improve this answer | |
  • 1
    $\begingroup$ So in other words, it is still there, but is often "ignored" by books and literature as they omit the $h()$ term ? $\endgroup$ – Xavier Bourret Sicotte Jun 1 '18 at 10:05
  • $\begingroup$ hmm, but is still doesn't explain why the coefficient n!∏mi=1xi! is omitted. @Xavier Bourret Sicotte have u found a satisfied explanation? $\endgroup$ – lnshi Dec 8 '18 at 2:32

I was trying to build the softmax regression from scratch, and got stuck here also: why all the posts just silently omit the coefficient $\frac{n!}{x_1! x_2! \dots x_k!}$ totally, after some research and thinking i share my understanding here:

In reality, we always use a special form of the multinomial distribution with $k>2$ and $n=1$, that is categorical distribution, and since $n=1$, then the coefficient $\frac{n!}{x_1! x_2! \dots x_k!}$ of the multinomial distribution's PMF will be always all 1, that is why we can omit it, more details:

$$ \begin{cases} \text{class 1 is chosen: } \frac{1!}{1! \cdot 0! \cdot 0! \dots 0!} = 1 \\ \text{class 2 is chosen: } \frac{1!}{0! \cdot 1! \cdot 0! \dots 0!} = 1 \\ \text{class 3 is chosen: } \frac{1!}{0! \cdot 0! \cdot 1! \dots 0!} = 1 \\ \vdots \end{cases} $$

For getting more details and a better full pic of understanding u can refer to my post GLM and exponential family distributions -> Why the PMF has no coefficient

Especially pay attention to understand what does the $x_i$ stand for in multinomial distribution from this section

| cite | improve this answer | |
  • 1
    $\begingroup$ Would you be better asking this as a new question since an answer cannot get an answer? $\endgroup$ – mdewey Dec 8 '18 at 14:47
  • $\begingroup$ @mdewey It reads like an effort to answer the question. It appears only to address a very special case of the question, though, so whether it is correct or useful may be for each reader to determine. $\endgroup$ – whuber Dec 8 '18 at 15:13
  • $\begingroup$ @mdewey i was trying to answer the question, not seeking any answer, since the accepted answer totally doesn't address the main concern of the question: why the coefficient is silently omitted, i just share my understanding to may help someone $\endgroup$ – lnshi Dec 9 '18 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.