Given $n$ probability density functions ($p_1$, ..., $p_n$) with known distributions, what are the ways of estimating the weights ($w_1$, ..., $w_n$) of these component distributions given a sample from a distribution $f$ which is some mixture of these probability density functions, i.e.

$f = \sum^n_{i=1} w_i p_i$.

I've found many methods looking at the seemingly harder problem, where the component distributions are unknown, but haven't had much luck with this simpler case.


The EM algorithm applies most straightforwardly to this simpler case, with the $Q(\cdot;\cdot)$ function defined by $$Q(\mathbf p,\mathbf p^{(t)})=\sum_{i=1}^N\sum_{j=1}^n \mathbb P_{\mathbf p^{(t)}}(Z_i=j|x_i)\log p_j(x_i)$$ As does a Bayesian approach putting a prior on $(p_1,\ldots,p_n)$, since the Gibbs sampler step are quite similar to the EM steps.


Here's the Bayesian solution I ended up using for a mixed discrete distribution. I believe this method is called the Richardson-Lucy algorithm. My original question was more general, but hopefully this will be helpful to someone.

The discrete probability density at a position $k$ in the mixed distribution $f$ given that it is made up of a mix of discrete density functions $p_i$, is given by

$$P(f_k) = \sum_i P(f_k|p_i)P(p_i),\tag{1}\label{eq1}$$

where $P(p_i) = w_i$ (the mixture weight from the original question). Baye's theorem can be stated as

$$P(f_k|p_i)w_i = P(p_i|f_k)P(f_k).\tag{2}\label{eq2}$$

From here we'd like to exclude the term $(p_i|f_k)$ and set up an iterative procedure to solve for $w_i$.

Similar to equation 1

$$w_i = \sum_k P(p_i|f_k)P(f_k),\tag{3}\label{eq3},$$

therefore, substituting equation 2 into 3, we get

$$w_i = \sum_k \frac{P(f_k|p_i)P(f_k)w_i}{P(f_k)}\tag{4}\label{eq4}.$$

Then, substituting equation 1 into 4, we get

$$w_i = \sum_k \frac{P(f_k|p_i)P(f_k)w_i}{\sum_j P(f_k|p_j)P(p_j)}\tag{5}\label{eq5}.$$

Given that the desired solution ($w_i$) appears on both sides of the equation we can set up an iterative process to estimate $w_i$ so that equation 5 becomes

$$w_{r+1,i} = w_{r,i} \sum_k \frac{P(f_k|p_i)P(f_k)}{\sum_j P(f_k|p_j)w_{r,i}}\tag{6}\label{eq6},$$


In my particular case I could directly measure $P(f_k|p_i)$ for each $p_i$ (the component distributions were known) so equation 6 was my solution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.