# weighted sum of posterior Dirichlet distributions

I have the following distribution:

$$q(\vec\theta) = \frac{\sum_k \alpha_k}{\sum_k \beta_k \alpha_k} (\sum_k\theta_k\beta_k) \frac{\Gamma(\sum_k \alpha_k)}{\prod_k\Gamma(\alpha_k)} \prod_k \theta_{k}^{\alpha_k-1} = \frac{\sum_k \alpha_k}{\sum_k \beta_k \alpha_k} (\sum_k\theta_k\beta_k) Dir(\vec\theta|\vec\alpha)$$

And I want to get the first moment of distribution $$q(\vec\theta)$$:

$$\mathop{\mathbb{E}}_q[\theta_k]=\int_\vec\theta\theta_kq(\vec\theta)d\vec\theta$$

My first question is about the limits of the integral. Since I am getting the expectation of $$q$$, should I integrate w.r.t. $$\vec\theta$$ or $$\theta_k$$. I mean $$\int_\vec\theta\theta_kq(\vec\theta)d\vec\theta$$ or $$\int_{\theta_k}\theta_kq(\vec\theta)d\theta_k$$.

After simplifying a little bit $$q(\vec\theta)$$, we get:

$$\mathop{\mathbb{E}}_q[\theta_k] = \frac{\sum_k \alpha_k}{\sum_k \beta_k \alpha_k} \int\theta_k(\sum_k\theta_k\beta_k) Dir(\vec\theta|\vec\alpha)d\theta_k = \frac{\sum_k \alpha_k}{\sum_k \beta_k \alpha_k} [\beta_1\int\theta_k\theta_1Dir(\vec\theta|\vec\alpha)d\theta_k + \beta_2\int\theta_k\theta_2Dir(\vec\theta|\vec\alpha)d\theta_k + ... + \beta_k\int\theta_k\theta_kDir(\vec\theta|\vec\alpha)d\theta_k ]$$

And this is starting to look as the mean of the Dirichlet Distribution. For instance, $$\mathop{\mathbb{E}}_p[X_k] = \frac{\phi_k}{\sum_k\phi_k}$$ for $$Dir(\vec X|\vec\phi)$$.

Any suggestions or help to 1) solve the integral and 2) verify over what to integrate (i.e. $$d\vec\theta$$ or $$d\theta_k$$)

BTW, I got confused with 2) because for $$Dir(\vec X|\vec\phi)$$ we have a mean $$\mathop{\mathbb{E}}_p[X_k] = \frac{\phi_k}{\sum_k\phi_k}$$

Or we can put the mean as: $$\mathop{\mathbb{E}}_p[X_k] = \int X_k \frac{\Gamma(\sum_k \phi_k)}{\prod_k\Gamma(\phi_k)}\prod_k X_k^{\phi_k-1} d\vec \theta = \frac{\phi_k}{\sum_k\phi_k}$$. However, the definition of expectation for a vector rv is: $$\mathop{\mathbb{E}}_p[X_k] = \int X_k p(\vec X) dX_k$$

• You can rewrite your distribution as a mixture of Dirichlets, therefore expressing the mean should not be an issue. Nov 20, 2018 at 16:31
• Ohh yeah. If I write q as a mixture of Dirichlet, I have k Dir similar to (Dir(\vec\theta|\phi)). Then I can solve the integral for another group of Dirichlet. So, basically, I need to change the parameters of my Dirichlet twice. After that, I have the mean. However, is it the same to integrate wrt d\theta_k and d\vec\theta. If so, I get an analytically form. Nov 20, 2018 at 18:29
• I have to integrate wrt to the vector for the definition of expectation. The other problem @Xi'an gave me a hint on how to solve it. So, can you answer the question and I can mark it as solved? Nov 25, 2018 at 3:58

In short, the posterior distribution is a mixture of Dirichlet distributions $$q(\theta)\propto\sum_{k=1}^K \beta_k\theta_k\text{Dir}(\theta;\alpha)\qquad\qquad(1)$$(where I removed the vector notation for $$\theta$$ and $$\alpha$$). Since the pdf of a Dirichlet $$\text{Dir}(\alpha)$$ distribution is$$\frac{1}{\mathrm{B}(\alpha)} \prod_{i=1}^K x_i^{\alpha_i - 1}$$where$$\mathrm{B}(\alpha) = \frac{\prod_{i=1}^K \Gamma(\alpha_i)}{\Gamma\bigl(\sum_{i=1}^K \alpha_i\bigr)}$$ the integral of the term $$\theta_k\text{Dir}(\theta;\alpha)$$ in (1) is $$\alpha_k/\mathbf{1}^T\alpha$$ and hence the normalising constant of this density $$q$$ is indeed$$\mathbf{1}^T\alpha\Big/\sum_{k=1}^K \beta_k\alpha_k$$ and the weight of the $$\text{Dir}(\theta;\alpha_1,\ldots,\alpha_k+1,\ldots,\alpha_K)$$ term in the mixture is $$\text{B}(\alpha_1,\ldots,\alpha_k+1,\ldots,\alpha_K))/\text{B}(\alpha)=\frac{\alpha_k\prod_{i=1}^K \Gamma(\alpha_i)}{\{\mathbf{1}^T\alpha\}\Gamma\bigl(\sum_{i=1}^K \alpha_i\bigr)}\Big/\frac{\prod_{i=1}^K \Gamma(\alpha_i)}{\Gamma\bigl(\sum_{i=1}^K \alpha_i\bigr)}=\frac{\alpha_k}{\{\mathbf{1}^T\alpha\}}$$hence$$q(\theta)=\sum_{k=1}^K \beta_k\alpha_k\text{Dir}(\theta;\alpha_1,\ldots,\alpha_k+1,\ldots,\alpha_K)\Big/\sum_{k=1}^K \beta_k\alpha_k$$The mean of a component $$\theta_j$$ of $$\theta$$ is thus given by $$\mathbb{E}_q[\theta_j]=\sum_{k=1}^K \beta_k\alpha_k \mathbb{E}[\theta_j| \alpha_1,\ldots,\alpha_k+1,\ldots,\alpha_K]\Big/\sum_{k=1}^K \beta_k\alpha_k$$namely $$\mathbb{E}_q[\theta_j]=\left\{\sum_{k\ne j} \beta_k\alpha_k\frac{\alpha_j}{\{\mathbf{1}^T\alpha\}+1}+\beta_j\alpha_j\frac{\alpha_j+1}{\{\mathbf{1}^T\alpha\}+1}\right\}\Big/\sum_{k=1}^K \beta_k\alpha_k$$ The specific questions about the integration over $$\theta$$ versus the integration over $$\theta_k$$ relate to the so-called "law of the unconscious statistician" in that the marginal of $$\theta_k$$ (which would be a mixture of Beta distributions in the current case) need not be derived for the expectation to be found.