# Upgrading weight parameters to random variable in Gaussian mixtures

In a Gaussian mixture model we model a density like:

$$p(\mathbf{x}|\pi,\mu,\sigma)=\sum \pi_i N(\mathbf{x}|\mu_i,\sigma_i)$$ [1]

where $$\pi,\mu$$ and $$\sigma$$ are parameters.

I would like to know if the following model is of any use / has a name.

Let's suppose that $$\mathbf{\pi} \sim Dirichlet(\mathbf{\alpha})$$, so that the weights of the Gaussian become a random variable. Than instead of [1] we have:

$$p(\mathbf{x}|\alpha,\mu,\sigma)=\int d\mathbf{\pi} p(\mathbf{\pi} |\alpha) \sum \pi_i N(\mathbf{x}|\mu_i,\sigma_i)$$ [2]

Does it make any sense to use [2] instead of [1] in some setting ? Does this operation make any sense in some setting ?

• Have a look at latent Dirichlet allocation, it’s similar. – A rural reader Mar 14 at 20:46
• Actually I was indeed studying LDA. I read this sentence on the article "It is important to distinguish LDA from a simple Dirichlet-multinomial clustering model. A classical clustering model would involve a two-level model in which a Dirichlet is sampled once for a corpus, a multinomial clustering variable is selected once for each document in the corpus, and a set of words are selected for the document conditional on the cluster variable." and I was trying to understand what they meant in this sentence... – Thomas Mar 15 at 8:16
• Maybe I should rewrite my question explaining what I am trying to understand... – Thomas Mar 15 at 8:17
• When setting a distribution like the Dirichlet distribution on the $\pi_i$'s, the goal is to facilitate the inference on the $\pi_i$'s, not to make them disappear. (This is the principle beyond Bayesian inference.) – Xi'an Mar 15 at 8:25
• @Xi'an thanks I will start first to understand your comment and than relate to the LDA sentence I wanted to understand. Could you please check my answer below ? If it is totally wrong I will remove it, otherwise please feel free to edit and correct. I would like to check my understanding... – Thomas Mar 15 at 13:33

Trying to answer Xi'an comment, to check if I am understanding his comment. As far as I am understanding he means that upgrading $$\pi$$ to random variable would have the meaning of modelling our uncertainty of $$\pi$$ but not the sampling process of the data points. I try to write this better for exercise.

In the standard setting, when we fit a Gaussian mixtures the $$\pi$$ are parameters. Let's us suppose $$\mu$$ and $$\sigma$$ fixed here and suppose we have $$M$$ samples. In the usual setting the EM algorithm finds the maximum likelyhood estimate:

$$\pi_{MLE}=argmax_{\pi} \ log L(\mathbf{x}|\pi)$$

Now instead of treating $$\pi$$ as a parameter we can upgrade it to a random variable. In order to do this we need to define its probability distribution and relation to $$x$$. In mathematical terms we need:

• $$P(\mathbf{x}|\pi)$$ , that we have ;

• A prior $$P(\pi)$$ ;

In order to do this we introduce a parameter $$\alpha$$ that defines the prior as a Dirchlet distribution and arrive at the graphical model:

which fully defines the joint $$P(\mathbf{x},\pi|\alpha)$$.

So what did we lose/gain with respect to MLE ?

1. Still all values of $$\pi$$ are possible/taken into account. If we condition on the value of $$\pi$$ we have $$P(\mathbf{x}|\pi,\alpha)=P(\mathbf{x}|\pi)$$. This is exactly the parametric likelyhood we had before ;

2. Still we can do inference of the value of $$\pi$$ by estimating the posterior $$P(\pi|\mathbf{x},\alpha)$$ ;

3. As a drawback, we introduced a parameter $$\alpha$$ which was not present before. Maybe it would make any sense to estimate $$\alpha$$ by maximizing the marginal likelyhood:

$$S(\alpha)=\int d\pi P(\pi|\alpha)P(\mathbf{x}|\pi)$$

even if this approach seems to suppose that the uncertainty in $$\pi$$ is related to the sampling process of $$\mathbf{x}$$. I have the impression that this way we would mix "uncertainties" and "sampling probabilites", but maybe these concepts are so interrelated that can be mixed together.

Actually this approach ( maximum marginal likelyhood ) in some sense is making the $$\pi$$ variables disappear the way described in the original message (weird...)

• This reflection upon my comment is essentially sound. It is correct that the inference will depend on the choice of the prior (hyper)parameter $\alpha$, that the MAP will differ from the MLE, and that $\alpha$ could be estimated from the marginal likelihood: the later technique is called empirical Bayes estimation and was proposed by Robbins in the 1950's. A description can be found in some Bayesian textbooks like Carlin and Louis (1992) or mine. – Xi'an Mar 15 at 13:47
• Great thanks for your feedback and suggestions – Thomas Mar 15 at 14:03