I want to have a mixture model like $\lambda \cdot P(s'|s, a) + (1- \lambda) \cdot P'(s'|s)$ where $P$ and $P'$ are conditional distributions and $\lambda \in [0,1]$ is a weight. I have two questions about the model.

  1. Is it true that the entire mixture model is still dependent on $a$ because its component $P$ is conditioned on $a$? Thus, can I write the mixture model as $P''(s'|s,a)$?
  2. This is a general question about any mixture model: should I interpret a mixture model as 1) first sampling a distribution based on given weights and then 2) sampling from the chosen distribution? Or, is it okay to interpret the model as we are just sampling once from the mixture model? I just started learning statistics, so I am not sure if this kind of interpretation even matters.

Thank you!

  • 2
    $\begingroup$ The main purpose of using a mixture model is you don’t know what groups/classes the observations belong to (or even how many groups there are). So foe question 2, you draw samples from the weighted component distributions proportional to those weights, but the interpretation is that the sample comes from “the mixture model”. $\endgroup$ – Earlien Jul 12 '20 at 5:20

Since the pdf of $S$ is $$p^{\prime\prime}(s'|s,a)=\lambda p(s'|a)+(1-\lambda)p'(s'|s)\tag{1}$$ a modification of either $a$ or $s$ modifies the pdf. The answer to question 1 is thus that the random variable $S'$ is dependent on the joint variable $(S,A)$.

A particular interpretation of the structure of density (1) allows for a further conditioning. If $Z$ is a Bernoulli $\mathcal B(\lambda)$ then $$p^{\prime\prime}(s'|s,a)=\mathbb P_\lambda(Z=1) p(s'|a)+\mathbb P_\lambda(Z=0)p'(s'|s)\tag{2}$$ which is identical to (1), and conditioning on $Z$ and $(S,A)$ leads to $$p^{\prime\prime}(s'|s,a,z)=\mathbb I_{z=1} p(s'|a)+\mathbb I_{z=0}p'(s'|s)\tag{3}$$ This means that, if $Z=1$, $S'$ is conditionally independent of $S$ and if $Z=0$, $S'$ is conditionally independent of $A$. This is however an interpretation in that $S'$ generated from (1) does not require the existence of $Z$. For instance, a PRG for (1) may use the inverse cdf transform $$\lambda F(s'|a)+(1-\lambda) F'(s'|s) = U\sim\mathcal U(0,1)$$ or an accept-reject algorithm, neither of which requires the construction and simulation of the latent variable $Z$.

  • $\begingroup$ Thanks! I guess then the term 'mixture model' itself assumes the existence of latent variables $Z$, which makes $s'$ independent of either $S$ or $A$ in your example. But the pdf of $S'$ does not assume any unless we explicitly write $Z$ in it. Then, we can think that we are directly sampling from a distribution that happens to have a form of $\lambda \cdot P(s'|a) + (1- \lambda) \cdot P'(s'|s)$, by which I think you mean $S'$ generated from (1) does not require the existence of $Z$. Is this correct? $\endgroup$ – Hunnam Jul 12 '20 at 15:37
  • $\begingroup$ I didn't understand when you explained the PRG part as I am not familiar with the concepts you used. But I will try to understand as I want to learn more! By the way, the distribution you wrote as $\mathcal{U}(0,1)$ is the uniform distribution? $\endgroup$ – Hunnam Jul 12 '20 at 15:44
  • $\begingroup$ Yes, $\mathcal U(01,)$ is a notation for the Uniform distribution. The last part of my answer is on the (pseudo-random) generation of variates from a mixture, in order to state that, given a mixture pdf or cdf, one does not require simulating the latent $Z$'s before simulating the $X$'s. $\endgroup$ – Xi'an Jul 12 '20 at 19:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.