1
$\begingroup$

I have an arbitrary number of independnet bivariate normal distributions with $\mu_i = [x_i,z_i]$ & $ \Sigma_i= \left(\begin{array}{cc} \sigma^2_{x_i} & \sigma^2_{x_i,z_i}\\\ \sigma^2_{x_i, z_i} & \sigma^2_{z_i} \end{array}\right) $

Where i is arbitrarily large

I want to take a linear combination of these bivariate normal distributions with weights $c = [c_1,...,c_i]$ where $\sum c_i = 1$ & $c_i >0$

Obviously, the linear combination of $\mu_{mixture} = [\sum c_ix_i,\sum c_iz_i]$

However, I am not sure about the linear combination of the variance-covariance matrix.

Does anyone know how I can calculate this pooled & weighted variance-covariance? Looking for the variance-covariance matrix for the mixture distribution.

Thanks so much!

$\endgroup$
7
  • $\begingroup$ Your question is ambiguous. Are you interested in properties of (a) the distribution of a linear combination of random variables having these distributions or (b) a mixture of these distributions? $\endgroup$
    – whuber
    Jul 8 at 19:33
  • 1
    $\begingroup$ @whuber Interested in the weighted mixture distribution of the i bivariate normal distributions $\endgroup$
    – CJR
    Jul 8 at 19:56
  • $\begingroup$ Please edit your post to state that clearly, because the answer that has been posted starts off with the other interpretation. $\endgroup$
    – whuber
    Jul 8 at 20:33
  • $\begingroup$ @whuber - thanks for catching that. I have edited the post $\endgroup$
    – CJR
    Jul 8 at 20:52
  • $\begingroup$ Your question has been asked and answered at stats.stackexchange.com/questions/16608 $\endgroup$
    – whuber
    Jul 8 at 22:02
2
$\begingroup$

Let $X_i\stackrel{\text{indep}}\sim\mathcal N(\mu_i,\Omega_i)$ and let $S = \sum_{i=1}^n c_iX_i$. A linear combination of independent Gaussians is Gaussian so we just need the mean and covariance. By linearity we have $$ \text E[S] = \sum_i c_i\mu_i $$ and by independence we have $$ \text{Var}[S] = \sum_i \text{Var}[c_iX_i] = \sum_i c_i^2 \Omega_i $$ so $$ S\sim\mathcal N\left(\sum_i c_i\mu_i, \sum_i c_i^2\Omega_i\right). $$ This applies no matter what the $c_i$ are and for any dimension of $X_i$.


The above part assumed $n < \infty$. If we have a countably infinite number of $X_i$ then whether or not the series $\sum_{i=1}^\infty c_i X_i$ converges depends on how the $c_i$, $\mu_i$, and $\Omega_i$ evolve and we can use Kolmogrov's three series theorem to understand when this happens.


I interpreted this to mean you wanted the distribution of a linear combination of Gaussians. If you meant a finite mixture of Gaussians then we can work it out in the following way. Let $f_i$ be the density of $X_i$ and let $S \sim \sum_{i=1}^n c_i f_i$ be the mixture distribution. You didn't state that $c_i \geq 0$ but I'll assume that so that this is a valid density. Then we have $$ \text E[S] = \int s \sum_i c_i f_i(s)\,\text ds = \sum_i c_i \text E[X_i] = \sum_i c_i \mu_i $$ as before, except now this represents a convex combination of the $\mu_i$ where that was not guaranteed before. I'll use $\mu_\text{mix} = \sum_i c_i\mu_i$ as the mixture mean.

For the variances we need $$ \text E[SS^T] = \int ss^T \sum_i c_i f_i(s)\,\text ds = \sum_i c_i \text E[X_iX_i^T] $$ so all together $$ \text{Var}[S] = \text E[SS^T] - (\text E S)(\text ES)^T \\ =\sum_i c_i \text E[X_iX_i^T] - \mu_\text{mix}\mu_\text{mix}^T $$ which is more complicated than $\sum_i c_i^2\Omega_i$

$\endgroup$
7
  • $\begingroup$ Thanks! This was my intuition & I appreciate the quick response. $\endgroup$
    – CJR
    Jul 8 at 19:24
  • $\begingroup$ @CJR for sure! glad this helped $\endgroup$
    – jld
    Jul 8 at 19:30
  • 1
    $\begingroup$ @CJR i just updated for the mixture case since it seems like maybe that's what you meant? $\endgroup$
    – jld
    Jul 8 at 20:16
  • $\begingroup$ Thanks for updating the post. I am still but unclear how the last line produces the variance-covariance matrix for the resulting mixture bivariate normal? If you could clarify that a bit more I'd really appreciate it. @jld $\endgroup$
    – CJR
    Jul 8 at 21:01
  • $\begingroup$ @CJR yeah sure. I'm using the result that for a random vector $S$ with mean vector $\mu$ the variance is $\text{Var}[S] = \text E[SS^T] - \mu\mu^T$. We know $\mu$ so the only remaining thing is the first term of $\text E[SS^T]$ and that's just the expected value of the 2x2 matrices $ss^T$ weighted by the mixture density $\endgroup$
    – jld
    Jul 8 at 21:45
0
$\begingroup$

Thanks to @jld & @whuber for their answeres. Both were super helpful as I tried to solve this problem. With continued research, I found the post which I'll link below. It has the same info that jld & whuber shared, but it helped me solve it so I wanted to link it here

https://math.stackexchange.com/questions/195911/calculation-of-the-covariance-of-gaussian-mixtures

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.