2
$\begingroup$

I've been reading up on Multinomial/Dirichlet priors and came across this note. I'm wondering why the normalising constant for the multinomial distribution drops out in the derivation of the joint distribution for the model (eq'n 3)

We have that the multinomial likelihood is equal to $\frac{N!}{\prod_{i=1}^Kn_i!}\prod_{i=1}^Kp_i^{n_i}$ with the factorial terms out the front forming a normalising constant. The dirichlet should be equal to $\frac{[\sum_i a_i]!}{\prod_{i=1}^Ka_i!}\prod_{i=1}^Kp_i^{a_i-1}$.

But I'm wondering why in the note I've linked the equation 3 gives that the posterior (prior Dirichlet multiplied by the likelihood multinomial) is $\prod_{i=1}^Kp_i^{n_i}.\prod_{i=1}^Kp_i^{a_i-1}.\frac{[\sum_i a_i]!}{\prod_{i=1}^Ka_i}$ which is clearly missing the normalising constant for the multinomial part of the joint distribution.

$\endgroup$
3
  • $\begingroup$ But wait, equation #3 isn't the posterior, it's joint distribution on $x$ and $\theta$. In order to get posterior on $\theta$ one needs to divide it by evidence $p(x) = \int p(x, \theta) d\theta$ (derived in eq. 10) $\endgroup$ Commented May 17, 2015 at 13:13
  • $\begingroup$ Sorry, fixed. I think my point still stands though doesn't it? As I think the joint distribution (in equation 3) should have a normalising constant from the multinomial? $\endgroup$
    – analystic
    Commented May 19, 2015 at 4:15
  • $\begingroup$ Goodness, I realised the title was mutlivariate instead of multinomial too. Not sure where my brain was that day. $\endgroup$
    – analystic
    Commented May 19, 2015 at 4:16

1 Answer 1

2
$\begingroup$

The formula in (3) refers to the probability of the actual ordered sequence $x$, not the probability of the "count variables" $n_1,\ldots,n_K$. The latter is distributed as a multinomial and requires the normalization factor. The normalization factor is the number of ways we can group $N$ items into $K$ buckets with sizes $n_1,\ldots,n_K$.

The probability of the ordered sequence $x$ is indeed just $\Pi_{i=1}^K p_i^{n_i}$ (think about tossing a die repeatedly.)

In fact, the author of the note is slightly incorrect when saying "we can summarize this sequence as a vector of $K$ count variables $n_1,\ldots,n_K$...", since the information about the order is lost when considering just the counts.

$\endgroup$
2
  • $\begingroup$ Thanks very much for clearing that up for me! I thought that might be the case but couldn't find any other resources on the full conjugate model to confirm it. $\endgroup$
    – analystic
    Commented May 26, 2015 at 0:53
  • $\begingroup$ I guess the confusion is that the multinomial doesn't code for information about order but the distribution there provided does. So it seems like the note is wrong in more than one way. $\endgroup$
    – analystic
    Commented May 26, 2015 at 0:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.