1
$\begingroup$

In a Gaussian mixture model we model a density like:

$p(\mathbf{x}|\pi,\mu,\sigma)=\sum \pi_i N(\mathbf{x}|\mu_i,\sigma_i)$ [1]

where $\pi,\mu$ and $\sigma$ are parameters.

I would like to know if the following model is of any use / has a name.

Let's suppose that $\mathbf{\pi} \sim Dirichlet(\mathbf{\alpha})$, so that the weights of the Gaussian become a random variable. Than instead of [1] we have:

$p(\mathbf{x}|\alpha,\mu,\sigma)=\int d\mathbf{\pi} p(\mathbf{\pi} |\alpha) \sum \pi_i N(\mathbf{x}|\mu_i,\sigma_i)$ [2]

Does it make any sense to use [2] instead of [1] in some setting ? Does this operation make any sense in some setting ?

$\endgroup$
5
  • 1
    $\begingroup$ Have a look at latent Dirichlet allocation, it’s similar. $\endgroup$ – A rural reader Mar 14 at 20:46
  • 1
    $\begingroup$ Actually I was indeed studying LDA. I read this sentence on the article "It is important to distinguish LDA from a simple Dirichlet-multinomial clustering model. A classical clustering model would involve a two-level model in which a Dirichlet is sampled once for a corpus, a multinomial clustering variable is selected once for each document in the corpus, and a set of words are selected for the document conditional on the cluster variable." and I was trying to understand what they meant in this sentence... $\endgroup$ – Thomas Mar 15 at 8:16
  • $\begingroup$ Maybe I should rewrite my question explaining what I am trying to understand... $\endgroup$ – Thomas Mar 15 at 8:17
  • $\begingroup$ When setting a distribution like the Dirichlet distribution on the $\pi_i$'s, the goal is to facilitate the inference on the $\pi_i$'s, not to make them disappear. (This is the principle beyond Bayesian inference.) $\endgroup$ – Xi'an Mar 15 at 8:25
  • $\begingroup$ @Xi'an thanks I will start first to understand your comment and than relate to the LDA sentence I wanted to understand. Could you please check my answer below ? If it is totally wrong I will remove it, otherwise please feel free to edit and correct. I would like to check my understanding... $\endgroup$ – Thomas Mar 15 at 13:33
1
$\begingroup$

Trying to answer Xi'an comment, to check if I am understanding his comment. As far as I am understanding he means that upgrading $\pi$ to random variable would have the meaning of modelling our uncertainty of $\pi$ but not the sampling process of the data points. I try to write this better for exercise.

In the standard setting, when we fit a Gaussian mixtures the $\pi$ are parameters. Let's us suppose $\mu$ and $\sigma$ fixed here and suppose we have $M$ samples. In the usual setting the EM algorithm finds the maximum likelyhood estimate:

$\pi_{MLE}=argmax_{\pi} \ log L(\mathbf{x}|\pi)$

Now instead of treating $\pi$ as a parameter we can upgrade it to a random variable. In order to do this we need to define its probability distribution and relation to $x$. In mathematical terms we need:

  • $P(\mathbf{x}|\pi)$ , that we have ;

  • A prior $P(\pi)$ ;

In order to do this we introduce a parameter $\alpha$ that defines the prior as a Dirchlet distribution and arrive at the graphical model:

enter image description here

which fully defines the joint $P(\mathbf{x},\pi|\alpha)$.

So what did we lose/gain with respect to MLE ?

  1. Still all values of $\pi$ are possible/taken into account. If we condition on the value of $\pi$ we have $P(\mathbf{x}|\pi,\alpha)=P(\mathbf{x}|\pi)$. This is exactly the parametric likelyhood we had before ;

  2. Still we can do inference of the value of $\pi$ by estimating the posterior $P(\pi|\mathbf{x},\alpha)$ ;

  3. As a drawback, we introduced a parameter $\alpha$ which was not present before. Maybe it would make any sense to estimate $\alpha$ by maximizing the marginal likelyhood:

$$S(\alpha)=\int d\pi P(\pi|\alpha)P(\mathbf{x}|\pi)$$

even if this approach seems to suppose that the uncertainty in $\pi$ is related to the sampling process of $\mathbf{x}$. I have the impression that this way we would mix "uncertainties" and "sampling probabilites", but maybe these concepts are so interrelated that can be mixed together.

Actually this approach ( maximum marginal likelyhood ) in some sense is making the $\pi$ variables disappear the way described in the original message (weird...)

$\endgroup$
2
  • 1
    $\begingroup$ This reflection upon my comment is essentially sound. It is correct that the inference will depend on the choice of the prior (hyper)parameter $\alpha$, that the MAP will differ from the MLE, and that $\alpha$ could be estimated from the marginal likelihood: the later technique is called empirical Bayes estimation and was proposed by Robbins in the 1950's. A description can be found in some Bayesian textbooks like Carlin and Louis (1992) or mine. $\endgroup$ – Xi'an Mar 15 at 13:47
  • $\begingroup$ Great thanks for your feedback and suggestions $\endgroup$ – Thomas Mar 15 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.