The mode in Dirichlet-Multinomial is

$$ \mathrm{Mode}(\pi_i) = \frac{\alpha_i + x_i - 1}{\sum_{j=1}^k (\alpha_j + x_j -1)} $$

  1. Could you point out how is it calculated please?
  2. What is the importance of -1 in "αi+xi−1" (I know that the mean/estimation formula is without "-1" so could you explain the effect that this "-1" added over the mean)?

Let $f(x_1,\dotsc,x_n)$ be the PDF of $\operatorname{Dir}(\alpha_1+1,\dotsc,\alpha_n+1)$. Then let

$$A=\log f(x_1,\dotsc,x_n)=\log(x_1^{\alpha_1}\dots x_n^{\alpha_n}) + C= \log(x_1^{\alpha_1})+\dots +\log(x_n^{\alpha_n}) + C$$ where $C$ is some constant.

We have this constraint on the variables $(x_1,\dotsc,x_n)$:

$$g(x_1,x_2,...,x_n) = \sum_i x_i =1 $$

Maximizing $A$ is the same as maximizing f. We introduce a Lagrange multiplier $\lambda$. Let the Lagrangian function be:

$$L(x_1,\dotsc,x_n,\lambda)= A-\lambda (g-c) = \log(x_1^{\alpha_1})+\dots +\log(x_n^{\alpha_n}) +C+ \lambda(x_1+\dots +x_n-1)$$ Taking the gradient of both sides gives:

$$d L(x_1,\dotsc,x_n) = \left({\alpha_1\over x_1} + \lambda\right)dx_1 + \dots \left({\alpha_n\over x_n} + \lambda\right)dx_n + (x_1+\dots +x_n-1)d\lambda$$

Solving for $dL=0$ gives $$\tag{1}x_i=-{\alpha_i \over \lambda} $$ and $$\sum_i x_i =1 \tag{2}$$ Apply the operator $\sum_i$ to $(1)$ taking into account $(2)$. This gives that $$\lambda = -\sum_i \alpha_i $$ which finally means that $$x_i=\frac{\alpha_i}{\sum_j \alpha_j} $$

Intuitive answer

The Dirichlet distribution represents an estimate of what categorical distribution produced some set of observations.

For example: If there's a scenario where there are three types of events:

  • event $1$ has been observed $5$ times

  • event $2$ has been observed $10$ times

  • event $3$ has been observed $7$ times,

then our "knowledge" about the probability distribution that produced those events is represented by the Dirichlet distribution $\operatorname{Dir}(5+1,10+1,7+1)$. It should almost be common sense that the most likely probabilities for events $1$, $2$ and $3$ should be $5\over5+10+7$, $10\over5+10+7$ and $7\over5+7+10$ respectively. Grouping them together as $(\frac5{22}, \frac{10}{22}, \frac{7}{22})$ then gives the mode of the Dirichlet distribution.

The above claim about the mode can be verified by solving a constrained optimisation problem. The optimisation needs to be constrained because the Dirichlet PDF is set to be zero outside those $(x_1, \dotsc, x_n)$ values for which $x_1+\dotsc+x_n=1$. This optimisation is easily done using Lagrange multipliers.

I've been trying to think about why the expression for the expected value is the same as the mode but the number of observations of each kind of event has increased by $1$. Proving it seems to be an exercise in integration. But I'd like to see an intuitive argument for that as well.

  • $\begingroup$ Thanks for the answer. Still I have a doubts here. Is there a standard way of calculating mode i.e. do we have to use a Lagrange multiplier always to calculate the mode? if no please let me know the other ways possible to calculate the mode. $\endgroup$ – Mosab Shaheen Feb 23 at 14:39
  • $\begingroup$ Lagrange multipliers are used because the PDF of the Dirichlet distribution is only non-zero when $x_1+\dots+x_n=1$. Therefore the optimisation needs to be constrained to those values. You wouldn't do it for a normal distribution for instance $\endgroup$ – man and laptop Feb 23 at 15:05
  • $\begingroup$ @MosabShaheen I've posted a more intuitive answer $\endgroup$ – man and laptop Feb 23 at 15:10
  • $\begingroup$ Thanks @manonlaptop. I saw for the normal distribution we can find it by setting the derivative to zero (to find the global maxima). May be it is not relevant to my question, but what is the standard name/idiom that is used to refer to that way (setting the derivative to zero)? $\endgroup$ – Mosab Shaheen Feb 23 at 16:01
  • $\begingroup$ @MosabShaheen Wikipedia says that it's called Fermat's Theorem or the Interior Extremum Theorem. It's not a term many people use. It's also likely to confuse people, as Fermat himself proved lots of theorem $\endgroup$ – man and laptop Feb 23 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.