1
$\begingroup$

The multiplication of multivariate Gaussian distributions defined over some parameter vector of a given dimension can be achieved by the following. Assuming that the Gaussian is parametrized by the precision matrix ($\Lambda$) and mean ($\mu$). We can compute the new precision as:

$$ \Lambda = \Lambda_1 + \Lambda_2 $$

Also, the new mean can be computed using the identity:

$$ \Lambda \mu = \Lambda_1 \mu_1 + \Lambda_2 \mu_2 $$

So, this is very convenient. However, I have a situation where I need to perform such a multiplication where one of the Gaussian distribution depends on $n$ number of variables. However, the second distribution depends on a subset of these $n$ variables. So, to perform this multiplication, I need to somehow make the second distribution also depend on all the $n$ variables, though in reality only a subset of these $n$ variables are affecting the underlying distribution.

The exact situation is that I have a posterior distribution $q(\theta)$ where $\theta$ is n-dimensional and I have the likelihood of the following form:

$$ L(\theta) = \prod_{i=1}^{k}L(\theta_i) $$

Here, $L(\theta_i)$ is a subset of $n$ variables and $L(\theta_i)$ is independent of $L(\theta_j)$ for $i \neq j$. I need to multiply $q(\theta)$ with a $L(\theta_i)$. So I somehow need to express this $L(\theta_i)$ as a function of $\theta$. Each $L(\theta_i)$ is a 3-dimensional normal distribution i.e. $\theta_i$ is 3-dimensional.

Does anyone know how this can be done?

$\endgroup$
4
  • $\begingroup$ I found the description a little hard to follow so I am not certain I am responding to the question you're actually asking. Don't you just need to integrate out the variables (dimensions) that aren't of immediate interest? $\endgroup$
    – Glen_b
    Mar 23, 2014 at 0:35
  • $\begingroup$ Probably not. I will edit the description as well. So basically what is going on is that I have a prior over n-dimensions and I have likelihood terms which factorise out in the usual way and each factor is a function of 3 variables. I am trying to use Expectation propagation to estimate the posterior distribution and so I need to multiply and divide Gaussians and there is a dimension mismatch. So, I need to somehow be able to represent this 3-dimensional Gaussian to be a function of n-dimensions to be able to multiply with the prior term and am struggling with that. $\endgroup$
    – Luca
    Mar 23, 2014 at 1:00
  • $\begingroup$ How are you determining the subset of n for the second component distribution? $\endgroup$ Mar 25, 2014 at 20:37
  • $\begingroup$ Each subset is a 3-dimensional Gaussian corresponding to a 3D spatial location. So the distribution is over $n \times 3$ parameters and the likelihood can be written as a product of $n$ terms i.e. $L(\theta) = \prod_{i}^{n} L(\theta_i)$ where each $L(\theta_i)$ is a 3D Gaussian. So each of these smaller subsets are independent of each other. However, the full distribution is over $\theta$ which is $3n$, so I need to somehow make these 3D Gaussians a function of $3n$ variables where the other $3n -3$ variables do not affect it. $\endgroup$
    – Luca
    Mar 25, 2014 at 20:51

3 Answers 3

1
+100
$\begingroup$

You might want to add into the question the comment which says that your likelihood $L(\theta) = \prod_{i=1}^n L(\theta_i)$ where each $\theta_i$ is 3-dimensional and $L(\theta_i)$ is 3D Gaussian.

So, if I understood correctly, your observation model is $y_i|\theta\sim N(\theta_i, \Sigma_i)$, where $y_i,\theta_i\in R^3$ and $\Sigma_i\in R^{3\times3}$ are, and $y_i$:s are conditionally independent. This can be equivalently written as a $3n$-dimensional normal distribution \begin{equation} y \sim N(\theta,\Sigma) \end{equation} where $y,\theta$ are 3n-dimensional vectors (just the $y_i$s and $\theta_i$s stacked together), and the joint covariance matrix $\Sigma$ is a block diagonal matrix where the elements corresponding to components belonging to the same $i$ are taken from the $\Sigma_i$s, and other elements of $\Sigma$ are 0.

To prove this, note that any linear combination of $y$ is a sum of linear combinations of $y_i$s, i.e., a sum of independent Gaussians, thus Gaussian. By definition, this implies that $y$ is $3n$-dimensional multivariate Gaussian. Then, to deduce the parameters, note that the means of $y_{k,i}$, and covariances of $y_{k,i},y_{l,i}$ are not impacted by taking the other variables into account, and based on the independence, covariances $Cov(y_{k,i},y_{l,j})$ must be zero for $i\neq j$.

$\endgroup$
0
0
$\begingroup$

ok, I am slightly confused. So, imagine for simplicity we have 4 parameters divided into 2 bivariate Gaussians. So, imagine the precision matrix for the first likelihood term is a 2x2 matrix with the terms

$$ \begin{pmatrix} a_{1,1} & a_{1,2} \\ a_{2,1} & a_{2,2} \end{pmatrix} $$

Similarly, the precision matrix for the second likelihood term is

$$ \begin{pmatrix} b_{1,1} & b_{1,2} \\ b_{2,1} & b_{2,2} \end{pmatrix} $$

Now, are you saying that if I want to represent either of the likelihood term as a 4-dimensional Gaussian, they will have the precision matrix as:

$$ \begin{pmatrix} a_{1,1} & a_{1,2} & 0 & 0 \\ a_{2,1} & a_{2,2} & 0 & 0 \\ 0 & 0 & b_{1,1} & b_{1,2} \\ 0 & 0 & b_{2,1} & b_{2,2} \end{pmatrix} $$

So I have the same precision matrix regardless of whether I want to represent likelihood term 1 or likelihood term 2. However, I am confused as to how both the terms can have the same precision matrix? Because if I just multiply this by itself, all the precision components which are non zero would change, even though it is equivalent to either multiplying by just one of the likelihood term.

$\endgroup$
2
  • $\begingroup$ No, sorry, I misunderstood the question+comments a bit and thought you don't need the separate likelihood terms. $\endgroup$ Mar 26, 2014 at 9:58
  • $\begingroup$ Ahhhh ok...I will edit the question now. It seems it is confusing! $\endgroup$
    – Luca
    Mar 26, 2014 at 11:13
0
$\begingroup$

Ok, it seems it is very simple. The precision matrix can be set to zeros for rows (i.e. uniform distribution) which do not affect the given likelihood term. Similarly, the $\Lambda \mu$ variable will have zero entries for variables not affecting the likelihood term.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.