3
$\begingroup$

I have derived the mean and variance of $\mu_j$ in Dirichlet distribution $\text{Dir}(\mu_1, \cdots, \mu_K|\alpha_1, \cdots, \alpha_k)$.

On https://en.wikipedia.org/wiki/Dirichlet_distribution, it also shows that

$$\mathbb{E}\left[\ln [\mu_j]\right] = \psi(\alpha_j) - \psi(\alpha_0)$$

where

  • $\alpha_0 = \sum_{k=1}^K \alpha_k$, and
  • $\psi(\alpha) = \frac{d}{d \alpha} \ln \Gamma(\alpha)$, the digamma function.

Can anyone provide hints or suggestion on how $\mathbb{E}\left[\ln \mu_j\right]$ can be derived, please?

$\endgroup$
2

2 Answers 2

5
$\begingroup$

\begin{align} \mathbb{E}[\ln \mu_j] &= \int_0^1 \ln \mu_j \text{Dir}(\boldsymbol{\mu}|\boldsymbol{\alpha}) d\mu_j \\ &= \int_0^1 \ln \mu_j \text{Beta}(\alpha_j, \alpha_0 - \alpha_j) d\mu_j \\ &= \int_0^1 \ln \mu_j \frac{1}{\text{B}(\alpha_j, \alpha_0 - \alpha_j)} \mu_j^{\alpha_j - 1} (1 - \mu_j)^{\alpha_0 - \alpha_j - 1} d\mu_j \\ &= \frac{1}{\text{B}(\alpha_j, \alpha_0 - \alpha_j)} \int_0^1 \frac{d \mu_j^{\alpha_j - 1}}{d \alpha_j}(1 - \mu_j)^{\alpha_0 - \alpha_j - 1} d\mu_j \\ &= \frac{1}{\text{B}(\alpha_j, \alpha_0 - \alpha_j)} \frac{d}{d \alpha_j} \int_0^1 \mu_j^{\alpha_j - 1} (1 - \mu_j)^{\alpha_0 - \alpha_j - 1} d\mu_j \\ &= \frac{1}{\text{B}(\alpha_j, \alpha_0 - \alpha_j)} \frac{d \text{B}(\alpha_j, \alpha_0 - \alpha_j)}{d \alpha_j} \\ &= \frac{d}{\alpha_j} \ln \text{B}(\alpha_j, \alpha_0 - \alpha_j) \\ &= \frac{d}{\alpha_j} \ln \frac{\Gamma(\alpha_j) \Gamma(\alpha_0 - \alpha_j)} {\Gamma(\alpha_0)} \\ &= \frac{d}{\alpha_j} \bigg( \ln \Gamma(\alpha_j) + \ln \Gamma(\alpha_0 - \alpha_j) - \ln \Gamma(\alpha_0) \bigg ) \\ &= \frac{d}{\alpha_j} \ln \Gamma(\alpha_j) - \frac{d}{\alpha_j}\ln \Gamma(\alpha_0) \\ &= \frac{d}{\alpha_j} \ln \Gamma(\alpha_j) - \frac{d}{\alpha_0}\ln \Gamma(\alpha_0) \\ &= \psi(\alpha_j) - \psi(\alpha_0) \end{align}

Note:

  • In the 4th equality, we used the fact $\frac{d}{dx} a^x = a^x \ln a$ as shown below.
  • In the 4th last and 2nd last equalities, when taking derivative wrt. $\alpha_j$, $\alpha_0 - \alpha_j$ is considered a constant, $\alpha_0$ is NOT a constant, so
    • $\frac{d}{\alpha_j} \ln(\alpha_0 - \alpha_j) = 0$ (4th last equality), and
    • $\frac{d}{\alpha_j}\ln \Gamma(\alpha_0) = \frac{d}{\alpha_j + (\alpha_0 - \alpha_j)}\ln \Gamma(\alpha_0) = \frac{d}{\alpha_0}\ln \Gamma(\alpha_0)$ (2nd last equality).
  • $\psi(x) \equiv \frac{d}{dx} \ln \Gamma(x) $ is called the digamma function.
$\endgroup$
0
$\begingroup$

The answer from @zxyue is fantastic.

However, it omitted the process how the 1st equation $\mathbb{E}[\ln\mu_j]=\int_0^1 \ln \mu_j \text{Dir}(\boldsymbol{\mu}|\boldsymbol{\alpha}) d\mu_j$ is achieved. This answer is an supplementary answer to @zxyue's answer.

According to the definition of Expectation: $\mathbb{E}[X]=\int_\mathbb{R}xf(x)dx$. Then $$ \begin{align} \mathbb{E}[\ln\mu_j] &=\int\ln\mu_jf(\boldsymbol{\mu})d\boldsymbol{\mu}\\ &=\int\dots\int\ln\mu_jf(\boldsymbol{\mu})d\mu_1\dots d\mu_K\\ &=\int\dots\int\int\int\dots\int\ln\mu_jf(\boldsymbol{\mu})d\mu_1\dots d\mu_{j-1}d\mu_jd\mu_{j+1}\dots d\mu_K \end{align} $$ Move integral $\int\dots d\mu_j$ to the most outside and set its interval as [0,1] $$ \mathbb{E}[\ln\mu_j] =\int_0^1\int_0^{1-\sum_{k=1}^{\mathbb{K}-1}{\mu_k}}\dots\int_0^{1-\sum_{k=1}^{j}{\mu_k}}\int_0^{1-\mu_j-\sum_{k=1}^{j-2}\mu_k}\dots\int_0^{1-\mu_j}\ln\mu_jf(\boldsymbol{\mu})d\mu_1\dots d\mu_{j-1}d\mu_{j+1}\dots d\mu_Kd\mu_j $$ Move $ln\mu_j$ out of the core integral then to the outermost integral, since $ln\mu_j$ is independent of $\mu_1\dots\mu_{j-1},\mu_{j+1}\dots\mu_K$ thus could be treated as a constant $$ \begin{align} \mathbb{E}[\ln\mu_j] &=\int_0^1\ln\mu_j\int_0^{1-\sum_{k=1}^{\mathbb{K}-1}{\mu_k}}\dots\int_0^{1-\sum_{k=1}^{j}{\mu_k}}\int_0^{1-\mu_j-\sum_{k=1}^{j-2}\mu_k}\dots\int_0^{1-\mu_j}f(\boldsymbol{\mu})d\mu_1\dots d\mu_{j-1}d\mu_{j+1}\dots d\mu_Kd\mu_j\\ &=\int_0^1\ln\mu_jg(\boldsymbol{\mu})d\mu_j \end{align} $$ Where $g(\boldsymbol{\mu})$ is $\int_0^{1-\sum_{k=1}^{\mathbb{K}-1}{\mu_k}}\dots\int_0^{1-\sum_{k=1}^{j}{\mu_k}}\int_0^{1-\mu_j-\sum_{k=1}^{j-2}\mu_k}\dots\int_0^{1-\mu_j}f(\boldsymbol{\mu})d\mu_1\dots d\mu_{j-1}d\mu_{j+1}\dots d\mu_K$

By observation, we could easily realize that $g(\boldsymbol{\mu})$ is the marginal distribution of variable ${M_j}$. That is to say $$ \begin{align} g(\boldsymbol{\mu})&=\int_0^{1-\sum_{k=1}^{\mathbb{K}-1}{\mu_k}}\dots\int_0^{1-\sum_{k=1}^{j}{\mu_k}}\int_0^{1-\mu_j-\sum_{k=1}^{j-2}\mu_k}\dots\int_0^{1-\mu_j}f(\boldsymbol{\mu})d\mu_1\dots d\mu_{j-1}d\mu_{j+1}\dots d\mu_K\\ &=f_{M_j}(\mu_j) \end{align} $$ As we known, $\boldsymbol{\mu}$ follows Dirichlet Distribution $\text{Dir}(\boldsymbol{\mu}|\boldsymbol{\alpha})$. Dirichlet Distribution is a multivariate version of Beta Distribution. Intuitively, we could get that $f_{M_j}(\mu_j)$ is the probability density function of Beta Distribution $\text{Beta}(\alpha_j, \alpha_0 - \alpha_j)$, where $\alpha_0=\sum_{j=1}^{K}\alpha_j$. Then we could write $\mathbb{E}[\ln\mu_j]$ as follows $$ \begin{align} \mathbb{E}[\ln\mu_j] &=\int\ln\mu_jf(\boldsymbol{\mu})d\boldsymbol{\mu}\\ &= \int_0^1\ln\mu_jg(\boldsymbol{\mu})d\mu_j\\ &= \int_0^1\ln\mu_jf_{M_j}(\mu_j)d\mu_j\\ &= \int_0^1 \ln \mu_j \text{Beta}(\alpha_j, \alpha_0 - \alpha_j) d\mu_j \end{align} $$ Which is now consistent with the 2nd equation from @zxyue's answer.

For a formal derivation of the marginal distribution of Dirichlet distribution, please refer the answer from question Find marginal distribution of 𝐾-variate Dirichlet

$\endgroup$
10
  • $\begingroup$ Why is there anything to show? The first equation is usually taken as a definition of expectation. When it's not, it is justified by LOTUS $\endgroup$
    – whuber
    Aug 19, 2021 at 16:08
  • $\begingroup$ If it is the definition of expectation and if it is with respect to its joint probability Dirichlet distribution ( as shown in the first equation), then the differential of the variable should be vector $d\boldsymbol{\mu}$ rather than $d\mu_j$ $\endgroup$
    – chengxiz
    Aug 20, 2021 at 6:54
  • $\begingroup$ That is correct--but one rarely bothers to observe this, because the first step is to integrate over all the other variables, thereby reducing the problem to one involving the marginal distribution. Thus, your answer could be reduced to its last line, which is a link. $\endgroup$
    – whuber
    Aug 20, 2021 at 11:02
  • $\begingroup$ I am happy we agreed that your previous comment was wrong. And, even I spent so many words for my answer to articulate how this marginal distribution came, you, as an experienced teacher, still failed to observe that at first glance. That’s exactly why I would like to insist my style rather than reduce it to one link. But you conclusion is correct this time anyway. $\endgroup$
    – chengxiz
    Aug 20, 2021 at 11:32
  • 1
    $\begingroup$ I agree with @whuber that the first equation is taken from the definition of expectation. To get the expectation of $\ln \mu_j$, all other $\mu_k$s in $\text{Dir}(\boldsymbol{\mu}|\boldsymbol{\alpha})$ are treated as constants. $\endgroup$
    – zyxue
    Aug 21, 2021 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.