5
$\begingroup$

The mode in Dirichlet-Multinomial is

$$ \mathrm{Mode}(\pi_i) = \frac{\alpha_i + x_i - 1}{\sum_{j=1}^k (\alpha_j + x_j -1)} $$

  1. Could you point out how is it calculated please?
  2. What is the importance of -1 in "αi+xi−1" (I know that the mean/estimation formula is without "-1" so could you explain the effect that this "-1" added over the mean)?
$\endgroup$
3
$\begingroup$

Let $f(x_1,\dotsc,x_n)$ be the PDF of $\operatorname{Dir}(\alpha_1+1,\dotsc,\alpha_n+1)$. Then let

$$A=\log f(x_1,\dotsc,x_n)=\log(x_1^{\alpha_1}\dots x_n^{\alpha_n}) + C= \log(x_1^{\alpha_1})+\dots +\log(x_n^{\alpha_n}) + C$$ where $C$ is some constant.

We have this constraint on the variables $(x_1,\dotsc,x_n)$:

$$g(x_1,x_2,...,x_n) = \sum_i x_i =1 $$

Maximizing $A$ is the same as maximizing f. We introduce a Lagrange multiplier $\lambda$. Let the Lagrangian function be:

$$L(x_1,\dotsc,x_n,\lambda)= A-\lambda (g-c) = \log(x_1^{\alpha_1})+\dots +\log(x_n^{\alpha_n}) +C+ \lambda(x_1+\dots +x_n-1)$$ Taking the gradient of both sides gives:

$$d L(x_1,\dotsc,x_n) = \left({\alpha_1\over x_1} + \lambda\right)dx_1 + \dots \left({\alpha_n\over x_n} + \lambda\right)dx_n + (x_1+\dots +x_n-1)d\lambda$$

Solving for $dL=0$ gives $$\tag{1}x_i=-{\alpha_i \over \lambda} $$ and $$\sum_i x_i =1 \tag{2}$$ Apply the operator $\sum_i$ to $(1)$ taking into account $(2)$. This gives that $$\lambda = -\sum_i \alpha_i $$ which finally means that $$x_i=\frac{\alpha_i}{\sum_j \alpha_j} $$


Intuitive answer

The Dirichlet distribution represents an estimate of what categorical distribution produced some set of observations.

For example: If there's a scenario where there are three types of events:

  • event $1$ has been observed $5$ times

  • event $2$ has been observed $10$ times

  • event $3$ has been observed $7$ times,

then our "knowledge" about the probability distribution that produced those events is represented by the Dirichlet distribution $\operatorname{Dir}(5+1,10+1,7+1)$. It should almost be common sense that the most likely probabilities for events $1$, $2$ and $3$ should be $5\over5+10+7$, $10\over5+10+7$ and $7\over5+7+10$ respectively. Grouping them together as $(\frac5{22}, \frac{10}{22}, \frac{7}{22})$ then gives the mode of the Dirichlet distribution.

The above claim about the mode can be verified by solving a constrained optimisation problem. The optimisation needs to be constrained because the Dirichlet PDF is set to be zero outside those $(x_1, \dotsc, x_n)$ values for which $x_1+\dotsc+x_n=1$. This optimisation is easily done using Lagrange multipliers.

I've been trying to think about why the expression for the expected value is the same as the mode but the number of observations of each kind of event has increased by $1$. Proving it seems to be an exercise in integration. But I'd like to see an intuitive argument for that as well.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for the answer. Still I have a doubts here. Is there a standard way of calculating mode i.e. do we have to use a Lagrange multiplier always to calculate the mode? if no please let me know the other ways possible to calculate the mode. $\endgroup$ – Mosab Shaheen Feb 23 '19 at 14:39
  • $\begingroup$ Lagrange multipliers are used because the PDF of the Dirichlet distribution is only non-zero when $x_1+\dots+x_n=1$. Therefore the optimisation needs to be constrained to those values. You wouldn't do it for a normal distribution for instance $\endgroup$ – ogogmad Feb 23 '19 at 15:05
  • $\begingroup$ @MosabShaheen I've posted a more intuitive answer $\endgroup$ – ogogmad Feb 23 '19 at 15:10
  • $\begingroup$ Thanks @manonlaptop. I saw for the normal distribution we can find it by setting the derivative to zero (to find the global maxima). May be it is not relevant to my question, but what is the standard name/idiom that is used to refer to that way (setting the derivative to zero)? $\endgroup$ – Mosab Shaheen Feb 23 '19 at 16:01
  • 1
    $\begingroup$ @Tim thanks for sharing the link. If you have some information, could you point out why the expectation and the mode differ by "-1" in "αi+xi−1" (what meaning/effect it can carry) $\endgroup$ – Mosab Shaheen Feb 23 '19 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.