3
$\begingroup$

I have the following distribution:

$q(\vec\theta) = \frac{\sum_k \alpha_k}{\sum_k \beta_k \alpha_k} (\sum_k\theta_k\beta_k) \frac{\Gamma(\sum_k \alpha_k)}{\prod_k\Gamma(\alpha_k)} \prod_k \theta_{k}^{\alpha_k-1} = \frac{\sum_k \alpha_k}{\sum_k \beta_k \alpha_k} (\sum_k\theta_k\beta_k) Dir(\vec\theta|\vec\alpha)$

And I want to get the first moment of distribution $q(\vec\theta)$:

$\mathop{\mathbb{E}}_q[\theta_k]=\int_\vec\theta\theta_kq(\vec\theta)d\vec\theta$

My first question is about the limits of the integral. Since I am getting the expectation of $q$, should I integrate w.r.t. $\vec\theta$ or $\theta_k$. I mean $\int_\vec\theta\theta_kq(\vec\theta)d\vec\theta$ or $\int_{\theta_k}\theta_kq(\vec\theta)d\theta_k$.

After simplifying a little bit $q(\vec\theta)$, we get:

$\mathop{\mathbb{E}}_q[\theta_k] = \frac{\sum_k \alpha_k}{\sum_k \beta_k \alpha_k} \int\theta_k(\sum_k\theta_k\beta_k) Dir(\vec\theta|\vec\alpha)d\theta_k = \frac{\sum_k \alpha_k}{\sum_k \beta_k \alpha_k} [\beta_1\int\theta_k\theta_1Dir(\vec\theta|\vec\alpha)d\theta_k + \beta_2\int\theta_k\theta_2Dir(\vec\theta|\vec\alpha)d\theta_k + ... + \beta_k\int\theta_k\theta_kDir(\vec\theta|\vec\alpha)d\theta_k ]$

And this is starting to look as the mean of the Dirichlet Distribution. For instance, $\mathop{\mathbb{E}}_p[X_k] = \frac{\phi_k}{\sum_k\phi_k}$ for $Dir(\vec X|\vec\phi)$.

Any suggestions or help to 1) solve the integral and 2) verify over what to integrate (i.e. $d\vec\theta$ or $d\theta_k$)


BTW, I got confused with 2) because for $Dir(\vec X|\vec\phi)$ we have a mean $\mathop{\mathbb{E}}_p[X_k] = \frac{\phi_k}{\sum_k\phi_k}$

Or we can put the mean as: $\mathop{\mathbb{E}}_p[X_k] = \int X_k \frac{\Gamma(\sum_k \phi_k)}{\prod_k\Gamma(\phi_k)}\prod_k X_k^{\phi_k-1} d\vec \theta = \frac{\phi_k}{\sum_k\phi_k}$. However, the definition of expectation for a vector rv is: $\mathop{\mathbb{E}}_p[X_k] = \int X_k p(\vec X) dX_k$

$\endgroup$
3
  • $\begingroup$ You can rewrite your distribution as a mixture of Dirichlets, therefore expressing the mean should not be an issue. $\endgroup$
    – Xi'an
    Nov 20 '18 at 16:31
  • $\begingroup$ Ohh yeah. If I write q as a mixture of Dirichlet, I have k Dir similar to (Dir(\vec\theta|\phi)). Then I can solve the integral for another group of Dirichlet. So, basically, I need to change the parameters of my Dirichlet twice. After that, I have the mean. However, is it the same to integrate wrt d\theta_k and d\vec\theta. If so, I get an analytically form. $\endgroup$
    – c.uent
    Nov 20 '18 at 18:29
  • $\begingroup$ I have to integrate wrt to the vector for the definition of expectation. The other problem @Xi'an gave me a hint on how to solve it. So, can you answer the question and I can mark it as solved? $\endgroup$
    – c.uent
    Nov 25 '18 at 3:58
2
$\begingroup$

In short, the posterior distribution is a mixture of Dirichlet distributions $$q(\theta)\propto\sum_{k=1}^K \beta_k\theta_k\text{Dir}(\theta;\alpha)\qquad\qquad(1)$$(where I removed the vector notation for $\theta$ and $\alpha$). Since the pdf of a Dirichlet $\text{Dir}(\alpha)$ distribution is$$\frac{1}{\mathrm{B}(\alpha)} \prod_{i=1}^K x_i^{\alpha_i - 1}$$where$$\mathrm{B}(\alpha) = \frac{\prod_{i=1}^K \Gamma(\alpha_i)}{\Gamma\bigl(\sum_{i=1}^K \alpha_i\bigr)}$$ the integral of the term $\theta_k\text{Dir}(\theta;\alpha)$ in (1) is $\alpha_k/\mathbf{1}^T\alpha$ and hence the normalising constant of this density $q$ is indeed$$\mathbf{1}^T\alpha\Big/\sum_{k=1}^K \beta_k\alpha_k$$ and the weight of the $\text{Dir}(\theta;\alpha_1,\ldots,\alpha_k+1,\ldots,\alpha_K)$ term in the mixture is $$\text{B}(\alpha_1,\ldots,\alpha_k+1,\ldots,\alpha_K))/\text{B}(\alpha)=\frac{\alpha_k\prod_{i=1}^K \Gamma(\alpha_i)}{\{\mathbf{1}^T\alpha\}\Gamma\bigl(\sum_{i=1}^K \alpha_i\bigr)}\Big/\frac{\prod_{i=1}^K \Gamma(\alpha_i)}{\Gamma\bigl(\sum_{i=1}^K \alpha_i\bigr)}=\frac{\alpha_k}{\{\mathbf{1}^T\alpha\}}$$hence$$q(\theta)=\sum_{k=1}^K \beta_k\alpha_k\text{Dir}(\theta;\alpha_1,\ldots,\alpha_k+1,\ldots,\alpha_K)\Big/\sum_{k=1}^K \beta_k\alpha_k$$The mean of a component $\theta_j$ of $\theta$ is thus given by $$\mathbb{E}_q[\theta_j]=\sum_{k=1}^K \beta_k\alpha_k \mathbb{E}[\theta_j| \alpha_1,\ldots,\alpha_k+1,\ldots,\alpha_K]\Big/\sum_{k=1}^K \beta_k\alpha_k$$namely $$\mathbb{E}_q[\theta_j]=\left\{\sum_{k\ne j} \beta_k\alpha_k\frac{\alpha_j}{\{\mathbf{1}^T\alpha\}+1}+\beta_j\alpha_j\frac{\alpha_j+1}{\{\mathbf{1}^T\alpha\}+1}\right\}\Big/\sum_{k=1}^K \beta_k\alpha_k$$ The specific questions about the integration over $\theta$ versus the integration over $\theta_k$ relate to the so-called "law of the unconscious statistician" in that the marginal of $\theta_k$ (which would be a mixture of Beta distributions in the current case) need not be derived for the expectation to be found.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.