Let $$z$$ be an $$m$$ dimensional vector distributed as $$z \sim N(0,D)$$ where $$D$$ is some diagonal covariance matrix.

And let $$x$$ be some $$n$$ dimensional vector whose conditional distribution with $$z$$ is given as $$p(x|z) \sim N(Az, 0)$$ where $$A$$ is some $$n \times m$$ matrix. This might seem a little weird to have no covariance; originally the covariance was $$k \mathbb{I}$$ ($$k$$=const and $$\mathbb{I}$$ is identity matrix) but this specific questions concerns the limit $$k \rightarrow 0$$.

Is it possible to calculate the mean and covariance of the conditional distribution $$p(z|x)$$?

I believe there is a result on p111 of the book by Kevin Murphy that can help with this. However I am having trouble applying it. Denoting covariance matrices by $$C$$, it tells me that the desired covariance should be

$$C_{(z|x)} =C_z - C_{(z,x)} C_x^{-1} C_{(x,z)}$$

It is clear that $$C_z=0$$ but how can I calculate the other two terms, namely the covariance of the joint distribution and the covariance of $$x$$?

Thank you

• You may rephrase the title of your question to be more specific. Something like "How to compute covariance of joint distribution?" would be better and more helpful when navigating through the question list. – Manu H Apr 28 '19 at 10:31

If we start from $$\mu_{x|z}=\mu_x+C_{xz}C_z^{-1}(z-\mu_z)=Az$$, and $$\mu_z=0$$, we get $$\mu_x=0,C_{xz}=AD$$. Then, we can use this in $$C_{x|z}=kI=C_x-C_{xz}C_z^{-1}C_{zx}$$ to get that $$C_x=kI+ADA^T$$ (also, use the fact that $$C_{zx}=C_{xz}^T$$). Now, we know all we need to calculate $$C_{z|x}$$: $$C_{z|x}=C_z-C_{xz}^TC_x^{-1}C_{xz}=D-DA^T(kI+ADA^T)^{-1}AD$$
From here you can take the limit of $$k$$ to whatever value you want. Clearly $$C_z\neq 0$$, and $$C_{xz}$$ is just a portion of the joint distribution's covariance matrix, where $$C_{xz}^{ij}=\operatorname{cov}(X_i,Z_j)$$.
• Thanks. Aren't you missing a transpose in the answer though: $D-D^TA^T(ADA^T)^{-1}AD = D-D^TA^T(A^T)^{-1}D^{-1}A^{-1}AD =D-D^TD^{-1}D=D-D^T$. – user11128 Apr 29 '19 at 0:53
• And, in fact, if D is diagonal, $D-D^T=0$. Right? – user11128 Apr 29 '19 at 0:55
• That’s because $D=D^T$, and you can’t inverse $A$; it’s not square. – gunes Apr 29 '19 at 3:47
• Ah right, of course. Thanks a lot! So in the $k \rightarrow 0$ limit, I can't simplify beyond dropping the $kI$ term in your original answer? – user11128 Apr 29 '19 at 9:38