Given a sample $X_m$ from each of $M$ distributions $f_m$ which are all mixtures of the same $C$ unknown distributions $g_c$ but with differing mixture coefficients $\alpha_{mc}$, (when and) how can I determine the mixture coefficients and distributions?

$X_m \sim \sum_{c=1}^C \alpha_{mc} \cdot g_c(x)\quad$ where $\quad\sum_{c=1}^C\alpha_{mc} =1, x \in \mathbf{R}^p \quad\quad \forall m<M$

What I know (not much):

If I was given only one sample (M=1), this would simply be a clustering problem.

For the case in which x is one dimensional ($p=1$) and only two mixtures and two component are concerned (M=C=2) and one sample is from one component alone ($\alpha_{1,1}=1$), Patra & Sen (2016) provide bounds on $\alpha$ using the ECDF and the monotonicity constraint for the reconstruction of $G_2(x)$. However, the curse of dimensionality hits my extension of it to $p>1$ hard (ECDF will be 1/Nth most of the time).

I currently approach the problem by maximum likelihood based on the observed k-nearest neighbor (knn) distances. But even with gradient information the optimization (using R's optim) appears difficult (local minima of -log(L)). This optimisation problem is very similar to Non-negative matrix factorization (NMF) objectives and the problem itself similar to hyperspectral unmixing. But I failed applying either directly to my problem.

I have the strong feeling that this problem has been solved before and I just dont't know its name - do you?

  • 1
    $\begingroup$ I faced a similar problem, where in my case $M\gg{C}$, the samples $X_m$ were of decent size, and $p=1$. I actually just did gridded kernel-density estimates of the $M$ distributions, and then applied NMF directly. Even in this case the answer was typically under-constrained a bit. If you have large $p$, I think you may need more prior constraint on the form of the end-member distributions. $\endgroup$
    – GeoMatt22
    Dec 20 '16 at 2:40
  • $\begingroup$ @GeoMatt22 I actually have M=10, C=2 (at least that is what I think I have), N=5000 each, p=4. As my distributions are pretty smooth, KDE might indeed be a good idea, I guess (I need to read on KDE vs KNNDE). I assume you used KL divergence as loss for NMF? But how did you enforce the "mixture coefficients sum to 1 for each mixture" constrain? I did not find a way to do so in the R package NMF. $\endgroup$
    – jan-glx
    Dec 21 '16 at 0:47
  • $\begingroup$ My approach was in Matlab and far from rigorous. I used the default Frobenius (matrix-L2) loss. I just interpreted the coefficients as "weights", so normalized by the sum. (The end-member distributions are also not guaranteed to sum to one either. Or rather their "discrete integral", allowing for a cell-volume "$|dX|$" factor.) I had $p=1$, and my "KDE" was actually just histograms post-processed with a Gaussian blur. $\endgroup$
    – GeoMatt22
    Dec 21 '16 at 1:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.