Multiplying Gaussian distributions of different dimensions

The multiplication of multivariate Gaussian distributions defined over some parameter vector of a given dimension can be achieved by the following. Assuming that the Gaussian is parametrized by the precision matrix ($\Lambda$) and mean ($\mu$). We can compute the new precision as:

$$\Lambda = \Lambda_1 + \Lambda_2$$

Also, the new mean can be computed using the identity:

$$\Lambda \mu = \Lambda_1 \mu_1 + \Lambda_2 \mu_2$$

So, this is very convenient. However, I have a situation where I need to perform such a multiplication where one of the Gaussian distribution depends on $n$ number of variables. However, the second distribution depends on a subset of these $n$ variables. So, to perform this multiplication, I need to somehow make the second distribution also depend on all the $n$ variables, though in reality only a subset of these $n$ variables are affecting the underlying distribution.

The exact situation is that I have a posterior distribution $q(\theta)$ where $\theta$ is n-dimensional and I have the likelihood of the following form:

$$L(\theta) = \prod_{i=1}^{k}L(\theta_i)$$

Here, $L(\theta_i)$ is a subset of $n$ variables and $L(\theta_i)$ is independent of $L(\theta_j)$ for $i \neq j$. I need to multiply $q(\theta)$ with a $L(\theta_i)$. So I somehow need to express this $L(\theta_i)$ as a function of $\theta$. Each $L(\theta_i)$ is a 3-dimensional normal distribution i.e. $\theta_i$ is 3-dimensional.

Does anyone know how this can be done?

• I found the description a little hard to follow so I am not certain I am responding to the question you're actually asking. Don't you just need to integrate out the variables (dimensions) that aren't of immediate interest? Mar 23, 2014 at 0:35
• Probably not. I will edit the description as well. So basically what is going on is that I have a prior over n-dimensions and I have likelihood terms which factorise out in the usual way and each factor is a function of 3 variables. I am trying to use Expectation propagation to estimate the posterior distribution and so I need to multiply and divide Gaussians and there is a dimension mismatch. So, I need to somehow be able to represent this 3-dimensional Gaussian to be a function of n-dimensions to be able to multiply with the prior term and am struggling with that.
– Luca
Mar 23, 2014 at 1:00
• How are you determining the subset of n for the second component distribution? Mar 25, 2014 at 20:37
• Each subset is a 3-dimensional Gaussian corresponding to a 3D spatial location. So the distribution is over $n \times 3$ parameters and the likelihood can be written as a product of $n$ terms i.e. $L(\theta) = \prod_{i}^{n} L(\theta_i)$ where each $L(\theta_i)$ is a 3D Gaussian. So each of these smaller subsets are independent of each other. However, the full distribution is over $\theta$ which is $3n$, so I need to somehow make these 3D Gaussians a function of $3n$ variables where the other $3n -3$ variables do not affect it.
– Luca
Mar 25, 2014 at 20:51

You might want to add into the question the comment which says that your likelihood $L(\theta) = \prod_{i=1}^n L(\theta_i)$ where each $\theta_i$ is 3-dimensional and $L(\theta_i)$ is 3D Gaussian.

So, if I understood correctly, your observation model is $y_i|\theta\sim N(\theta_i, \Sigma_i)$, where $y_i,\theta_i\in R^3$ and $\Sigma_i\in R^{3\times3}$ are, and $y_i$:s are conditionally independent. This can be equivalently written as a $3n$-dimensional normal distribution $$y \sim N(\theta,\Sigma)$$ where $y,\theta$ are 3n-dimensional vectors (just the $y_i$s and $\theta_i$s stacked together), and the joint covariance matrix $\Sigma$ is a block diagonal matrix where the elements corresponding to components belonging to the same $i$ are taken from the $\Sigma_i$s, and other elements of $\Sigma$ are 0.

To prove this, note that any linear combination of $y$ is a sum of linear combinations of $y_i$s, i.e., a sum of independent Gaussians, thus Gaussian. By definition, this implies that $y$ is $3n$-dimensional multivariate Gaussian. Then, to deduce the parameters, note that the means of $y_{k,i}$, and covariances of $y_{k,i},y_{l,i}$ are not impacted by taking the other variables into account, and based on the independence, covariances $Cov(y_{k,i},y_{l,j})$ must be zero for $i\neq j$.

ok, I am slightly confused. So, imagine for simplicity we have 4 parameters divided into 2 bivariate Gaussians. So, imagine the precision matrix for the first likelihood term is a 2x2 matrix with the terms

$$\begin{pmatrix} a_{1,1} & a_{1,2} \\ a_{2,1} & a_{2,2} \end{pmatrix}$$

Similarly, the precision matrix for the second likelihood term is

$$\begin{pmatrix} b_{1,1} & b_{1,2} \\ b_{2,1} & b_{2,2} \end{pmatrix}$$

Now, are you saying that if I want to represent either of the likelihood term as a 4-dimensional Gaussian, they will have the precision matrix as:

$$\begin{pmatrix} a_{1,1} & a_{1,2} & 0 & 0 \\ a_{2,1} & a_{2,2} & 0 & 0 \\ 0 & 0 & b_{1,1} & b_{1,2} \\ 0 & 0 & b_{2,1} & b_{2,2} \end{pmatrix}$$

So I have the same precision matrix regardless of whether I want to represent likelihood term 1 or likelihood term 2. However, I am confused as to how both the terms can have the same precision matrix? Because if I just multiply this by itself, all the precision components which are non zero would change, even though it is equivalent to either multiplying by just one of the likelihood term.

• No, sorry, I misunderstood the question+comments a bit and thought you don't need the separate likelihood terms. Mar 26, 2014 at 9:58
• Ahhhh ok...I will edit the question now. It seems it is confusing!
– Luca
Mar 26, 2014 at 11:13

Ok, it seems it is very simple. The precision matrix can be set to zeros for rows (i.e. uniform distribution) which do not affect the given likelihood term. Similarly, the $\Lambda \mu$ variable will have zero entries for variables not affecting the likelihood term.