3
$\begingroup$

Is sampling all distributions n times and then talking out i numbers from each sample, where i is probability of that distribution * n, wrong?

Suppose $$ 0.3\!\times\mathcal{N}(0,1)\; + \;0.5\!\times\mathcal{N}(10,1)\; + \;0.2\!\times\mathcal{N}(3,.1) $$ being my problem to sample 100 numbers. Should I take 100 or say 1000 samples each of the three normal distributions and then take 30, 50 and 20 respectively, randomly out of them?

Or going by another approach, Should I take 30, 50 and 20 random samples respectively, directly from the three distributions?

The correct algorithm seems to be:

  • generate a number, say k according to the probabilities of all distributions which corresponds to a particular k-th one.
  • generate a number from the above k-th distribution.

Repeat for N numbers. See it at sampling from a mixture of two Gamma distributions .

Are all these three approaches same?

fwiw, I am using python and I am not familiar with R. And the reason I am asking this question is this comment:

[..]The notation suggests that to sample, you need to sample all three normals and weigh the results by those coefficients which would obviously not be correct. [..]

here

Improve this question
$\endgroup$
4
  • $\begingroup$ If you are seeking for clarification of a comment of some random user of the internet, better just ask him directly in comment. What is described is not "notation" but direct formula for the mixture distribution, hard to say why someone "doesn't like it". Your idea how to sample form the mixture is almost correct stats.stackexchange.com/a/243398/35989 $\endgroup$
    – Tim
    Commented Oct 12, 2019 at 19:55
  • $\begingroup$ Umm no, any any of the two approaches there wrong? is the question here. I might post a comment under theirs too, but that is just background. $\endgroup$
    – user230053
    Commented Oct 12, 2019 at 19:56
  • $\begingroup$ See the link I posted. $\endgroup$
    – Tim
    Commented Oct 12, 2019 at 19:58
  • $\begingroup$ yes, I have come across that answer and was about to include it into this question too, to ease comparison. $\endgroup$
    – user230053
    Commented Oct 12, 2019 at 19:59

1 Answer 1

Reset to default
1
$\begingroup$

The quote you refer to

The notation suggests that to sample, you need to sample all three normals and weigh the results by those coefficients which would obviously not be correct.

seems to misunderstand the notation. Mixture distribution of $m$ $f_k$ components and mixing weights $\pi_k$ is defined as

$$ f(x) = \sum_{k=1}^m \pi_k \; f_k(x) $$

Weighting the distributions and weighting the values are not the same things. Moreover, we are looking at the probability of observing $x$ according to each of the $f_k$ distributions, not weighted sum of three different random variables

$$ \pi_X \, f_X(x) + \pi_Y \, f_Y(y) + \pi_Z \, f_Z(z) $$

Drawing samples from three distributions and weighting them has nothing to do with mixture distribution. The notation does not suggest anything like this It is the opposite, we are summing the components because they are mutually exclusive.

So the correct way of thinking about mixture is that you first randomly choose the $k$-th component with probability $\pi_k$, and then draw a sample from this component according to the distribution $f_k$. Same happens in the algorithm for sampling from mixture distribution.

As about your proposed algorithm, it is not equivalent to the proper algorithm. If you needed to simulate 100 draws using a fair coin, you would not take 50 heads and 50 tails and shuffle them, this would not be a valid sample. If the probability of drawing heads is 0.5 this does not mean that in sample of size $n$ you would observe $n\times 0.5$ heads. It means that with $n$ large enough you would see approximately that many heads. Same applies to mixtures, you need to draw the components randomly.

Sorry but I don't follow what you mean by your first algorithm, where you want to draw 1000 samples to obtain 100 samples from the mixture.

Improve this answer
$\endgroup$
5
  • $\begingroup$ the first algo is about taking 1000 or 100 or any number bigger than 30 of first distribution , and then taking 30 out of them. then repeated for others to make total 100. I admit, it was based on false deductions. Ignore it. you may please remove it from both the question and answer. $\endgroup$
    – user230053
    Commented Oct 12, 2019 at 21:59
  • $\begingroup$ pastebin.com/vZhq2xQN hope I've got the code correct. $\endgroup$
    – user230053
    Commented Oct 12, 2019 at 22:05
  • 1
    $\begingroup$ @ankii code seems to be fine. $\endgroup$
    – Tim
    Commented Oct 12, 2019 at 22:11
  • $\begingroup$ I totally get that, but I have seen the notation for a mixture of three Gaussians in the OP mislead many students into thinking it concerns a linear combination of random variables instead of mixture. And my opinion is that the notation is indeed suggestive of being wrongly interpreted. My gripe is therefore only with the notation $p_1\times N(0,1)+p_2\times N(10,1)+p_3\times N(3,0.1)$ which I would rather change to something like Mix($N(0,1)$,$N(10,1)$,$N(3,0.1)$; $p_1$, $p_2$, $p_3$) $\endgroup$
    – StijnDeVuyst
    Commented Oct 14, 2019 at 15:46
  • $\begingroup$ @StijnDeVuyst you can use whatever notation you want but "Mix" tells you nothing about what the distribution is, while $p_1 N(\mu_1, \sigma_1) + p_2 N(\mu_2, \sigma_2)$ is the probability density function that defines the mixture distribution. You cannot change definition with an invented name. $\endgroup$
    – Tim
    Commented Oct 14, 2019 at 16:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.