2
$\begingroup$

I've been reading up on Multinomial/Dirichlet priors and came across this note. I'm wondering why the normalising constant for the multinomial distribution drops out in the derivation of the joint distribution for the model (eq'n 3)

We have that the multinomial likelihood is equal to $\frac{N!}{\prod_{i=1}^Kn_i!}\prod_{i=1}^Kp_i^{n_i}$ with the factorial terms out the front forming a normalising constant. The dirichlet should be equal to $\frac{[\sum_i a_i]!}{\prod_{i=1}^Ka_i!}\prod_{i=1}^Kp_i^{a_i-1}$.

But I'm wondering why in the note I've linked the equation 3 gives that the posterior (prior Dirichlet multiplied by the likelihood multinomial) is $\prod_{i=1}^Kp_i^{n_i}.\prod_{i=1}^Kp_i^{a_i-1}.\frac{[\sum_i a_i]!}{\prod_{i=1}^Ka_i}$ which is clearly missing the normalising constant for the multinomial part of the joint distribution.

$\endgroup$
3
  • $\begingroup$ But wait, equation #3 isn't the posterior, it's joint distribution on $x$ and $\theta$. In order to get posterior on $\theta$ one needs to divide it by evidence $p(x) = \int p(x, \theta) d\theta$ (derived in eq. 10) $\endgroup$ May 17 '15 at 13:13
  • $\begingroup$ Sorry, fixed. I think my point still stands though doesn't it? As I think the joint distribution (in equation 3) should have a normalising constant from the multinomial? $\endgroup$
    – analystic
    May 19 '15 at 4:15
  • $\begingroup$ Goodness, I realised the title was mutlivariate instead of multinomial too. Not sure where my brain was that day. $\endgroup$
    – analystic
    May 19 '15 at 4:16
2
$\begingroup$

The formula in (3) refers to the probability of the actual ordered sequence $x$, not the probability of the "count variables" $n_1,\ldots,n_K$. The latter is distributed as a multinomial and requires the normalization factor. The normalization factor is the number of ways we can group $N$ items into $K$ buckets with sizes $n_1,\ldots,n_K$.

The probability of the ordered sequence $x$ is indeed just $\Pi_{i=1}^K p_i^{n_i}$ (think about tossing a die repeatedly.)

In fact, the author of the note is slightly incorrect when saying "we can summarize this sequence as a vector of $K$ count variables $n_1,\ldots,n_K$...", since the information about the order is lost when considering just the counts.

$\endgroup$
2
  • $\begingroup$ Thanks very much for clearing that up for me! I thought that might be the case but couldn't find any other resources on the full conjugate model to confirm it. $\endgroup$
    – analystic
    May 26 '15 at 0:53
  • $\begingroup$ I guess the confusion is that the multinomial doesn't code for information about order but the distribution there provided does. So it seems like the note is wrong in more than one way. $\endgroup$
    – analystic
    May 26 '15 at 0:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.