1
$\begingroup$

Let's say the density of $Y$ is given by $p(y)=\frac{1}{L}\sum^L_{i=1}N(y\mid \mu_i, \Sigma_i)$, where $N(y \mid \mu_i, \Sigma_i)$ is the multivariate normal density evaluated at $y$, with known $L,\mu_i,\Sigma_i$.

  1. If I wanted to simulate from $p(y),$ could I just take a multinomial with $L$ probabilities, and then simulate from the respective $i$-th category?
  2. If we have a very big $L,$ something like $20,000.$ What's then a good way to sample from that mixture? When generating a sample from a mixture $\sum w_i f_i$, with a given size (N), why not simply take $N_i$ draws from each density $f_i$, such that $\frac{N_i}{N}=w_i$? Can we do it?
  3. Is there a closed formula representation for a mixture of multivariate normals?
$\endgroup$
1
$\begingroup$
  1. Yes, you can take a multinomial with $n=1$ and using a probability vector $p=\mathbb{1}_{1\times L}\frac{1}{L}$. Or use Categorical Distribution, which is actually a special case. Another simple method would be generating a uniform random variable $U$, in $[0,1)$, and choose the category $\lfloor LU \rfloor$ (if numbered from $0$). In non-equal mixture probabilities this is handled internally in random number generation via slight modification of the binary search idea.

  2. It shouldn't matter much when you have $L=20000$. But, you can also use the uniform RV approach to get rid of probability vector allocation as well. For your updated extension of Q2, as I've noted in the comments, when split sizes (elements coming from each mixture component) are assumed to be constant, you're not truly sampling the mixture distribution you wrote; even if it can emulate the mixture in large samples.

  3. To the best of my knowledge, there is none. I think, trying to write it in a closer form is like trying to write $e^a+e^b$ in closer form, in which I don't see any further simplification.

$\endgroup$
  • $\begingroup$ Gunes, thank you for your answer. I still have a question though. When generating a sample from a mixture $\sum w_i f_i$, with a given size (N), why not simply take $N_i$ draws from each density $f_i$, such that $\frac{N_i}{N}=w_i$? Can we do it? $\endgroup$ – An old man in the sea. Feb 12 at 17:56
  • $\begingroup$ @Anoldmaninthesea. But you’re then guaranteeing the mixture ratios in every sample. That’s biased. $\endgroup$ – gunes Feb 12 at 18:09
  • $\begingroup$ why do you say that's biased? In the limit, as sample size tends to infinity, that's what we'll get, right? $\endgroup$ – An old man in the sea. Feb 12 at 18:15
  • $\begingroup$ Oh, bad choice of word. It resembles the biased/unbiased estimation. However, yes, in the long run, your samples will better represent the mixture PDF, but note that you're not truly sampling it. Let's say you've $N=100$ and $2$ mixtures. True samples from this mixture would contain the sample split $[40,60]$ or $[45,55]$ for example, but in your way, you'll always get $[50,50]$. $\endgroup$ – gunes Feb 12 at 18:24
  • 1
    $\begingroup$ Thanks for everything ;) $\endgroup$ – An old man in the sea. Feb 12 at 19:02

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.