# Correlated random variables from mixture distributions

Let I have three random variables whose density is a mixture of two Normals with these parameters:

1. $$\mu_{1,1}=6.8$$, $$\mu_{1,2}=6.95$$, $$\sigma_{1,1}=0.065$$, $$\sigma_{1,2}=0.055$$ and $$\alpha_{1}=0.4$$
2. $$\mu_{2,1}=5.7$$, $$\mu_{2,2}=5.92$$, $$\sigma_{2,1}=0.08$$, $$\sigma_{2,2}=0.09$$ and $$\alpha_{2}=0.3$$
3. $$\mu_{3,1}=4.9$$, $$\mu_{3,2}=5.01$$, $$\sigma_{3,1}=0.04$$, $$\sigma_{3,2}=0.1$$ and $$\alpha_{3}=0.2$$

$$\alpha_{i}$$ is the weight of the first density for variable $$i$$.

Moreover, I know that those random variables have this correlation matrix (I know it's positive semi-definite, you can change it for numerical examples purposes):

$$\textbf{P}=\begin{bmatrix} 1 & 0.3 & -0.4 \\ 0.3 & 1 & -0.1 \\ -0.4 & -0.1 & 1 \end{bmatrix}$$

I would like to generate correlated random numbers from those mixtures. If you could provide R code, this would be a strong plus.

• I don't understand the distribution you're assuming. Aren't the above parameters suggestion you have a mixture of 3 distributions for two random variables? It looks like your full pdf would be given by: $\alpha_{1}N(x_{1};\mu_{1,1},\sigma_{1,1}) N(x_{2};\mu_{1,2},\sigma_{1,2})+\alpha_{2}N(x_{1};\mu_{2,1},\sigma_{2,1}) N(x_{2};\mu_{2,2},\sigma_{2,2})+\alpha_{3}N(x_{1};\mu_{3,1},\sigma_{3,1}) N(x_{2};\mu_{3,2},\sigma_{3,2})$ ? If it's a mixture of two normals, what's the third $\alpha$ for ? – gazza89 Sep 27 '18 at 10:49
• Sorry for the ambiguity, the first distribution is made up by two Normal densities: $\alpha_{1}N(\mu_{1,1},\sigma_{1,1})+(1-\alpha_{1})N(\mu_{1,2},\sigma_{1,2})$. The second random variable follows a distribution made up by two Normal densities: $\alpha_{2}N(\mu_{2,1},\sigma_{2,1})+(1-\alpha_{2})N(\mu_{2,2},\sigma_{2,2})$. And so forth. Actually my notation was not the simplest. – Lisa Ann Sep 27 '18 at 11:04
• In that case, why aren't all three variables mutually independent ? Where is the correlation coming from? – gazza89 Sep 27 '18 at 11:07
• Uhm... Let I have estimated those mixture parameters by an EM algorithm and by using separated samples. Then I have taken those separated samples and estimated their correlation somehow (e.g. rank). Isn't this theoretically consistent? – Lisa Ann Sep 27 '18 at 11:16
• If you assume the model you've assumed, you've baked in feature independence. Your EM algorithm will maximise the likelihood of your data given a model, but if you can see that your features are correlated, you should probably assume a different model and run EM. E.g. $\alpha _{1}N(\underline{x}; \underline{\mu}_{1}, \underline{\underline{\Sigma}}_{1} ) + (1-\alpha _{1})N(\underline{x}; \underline{\mu}_{2}, \underline{\underline{\Sigma}}_{2} )$, i.e. a superposition of multivariate Gaussians – gazza89 Sep 27 '18 at 12:52

You need to identify a three dimensional Normal mixture which global covariance matrix is Q. This means finding $$\mathbf{Q}_1$$ and $$\mathbf{Q}_2$$ such that $$\alpha(\mathbf{Q}_1+\mu_{\cdot 1}\mu_{\cdot 1}^\text{T})+ (1-\alpha)(\mathbf{Q}_2+\mu_{\cdot 1}\mu_{\cdot 2}^\text{T})= \mathbf{Q}+(\alpha\mu_{\cdot 1}+(1-\alpha)\mu_{\cdot 2})(\alpha\mu_{\cdot 1}+(1-\alpha)\mu_{\cdot 2})^\text{T})$$ The number of unknowns in this equation are 3 correlations in $$\mathbf{Q}_1$$ and 3 correlations in $$\mathbf{Q}_2$$, since the diagonal terms are given by the marginal Normals. Hence there is a range of choices, provided $$\mathbf{P}$$ is an achievable correlation matrix for a mixture.
Generating from a multivariate Normal mixture is straightforward: select the component with probability $$\alpha$$ versus $$1-\alpha$$ and generate the associated multivariate Normal.
• Could you please expand a bit? Some questions: (1) what would there be on the main diagonal of $\mathbf{Q}$? In the case of multivariate Normal there would be the variance; in the case of Normal mixture, it's hard to say what would there be on the diagonal. (2) Does a Gaussian Copula approach work like you said above? So I can apply the inverse Normal to the quantiles of the Normal mixtures and get a multivariate Normal with correlation $\mathbf{P}$. – Lisa Ann Sep 28 '18 at 9:39
• (1) the diagonal terms of $\mathbf{Q}_1$, $\mathbf{Q}_2$, $\mathbf{Q}$ are all variances, which are given for the mixture components and a consequence of the mixture representation for the resulting $\mathbf{Q}$ . – Xi'an Sep 28 '18 at 11:08