Conditional Density between three gaussians

Question

Suppose we have two independent standard normal variables $X$ and $Y$ and lets construct $Z$ such as $Z = \rho X + \sqrt{1-\rho^2 Y}$. Therefore, $Z$ is dependent on $X$ and $Y$. I want to find the distribution of $Z|X, Y$. However, when I try to generate the Multivariate Normal distribution from these three Gaussians, I obtain a singular covariance matrix. I read the Wikipedia article on the Multivariate Normal distribution and it mentions a degenerate case when the covariance matrix is singular, giving the following expression:

$f(\mathbf{x})= \left(\det\nolimits^*(2\pi\boldsymbol\Sigma)\right)^{-\frac{1}{2}}\, e^{ -\frac{1}{2}(\mathbf{x}-\boldsymbol\mu)^{{{\!\mathsf{T}}}} \boldsymbol\Sigma^+(\mathbf{x}-\boldsymbol\mu) }$

where $\boldsymbol\Sigma^+$ is the generalized inverse and $\text{det}*$ is the pseudo-determinant.

So, my question is: is it correct to use this distribution to find $Z|X,Y$? And in this case, can I still say that $Z|X,Y$ has continuous pdf of $\frac{f(x,y,z)}{f(x,y)}$? I lack some knowledge on this topic, so I would also be grateful for some references.

Answer 1

Since $X$ and $Y$ are independent, $(X, Y)$ have a joint bivariate normal distribution. Then you define $Z$ as a function of $X$ and $Y$ (I assume $\rho$ is a known constant). So the conditional distribution of $Z$ given that $X=x, Y=y$ is simply the constant $z=\rho x + \sqrt{1-\rho^2} y$.

So the joint distribution of $X, Y, Z$ is singular (see Multivariate normal with singular covariance, Can a multivariate distribution with a singular covariance matrix have a density function?, )

In your case the function defining $Z$ is a linear function, so a full description of the joint distribution can be given by the bivariate normal distribution of $X, Y$, and that linear function. If you replaced the linear function by some non-linear function, for example $f(x,y) = x + y^3$, then the distribution would still be singular (the probability mass would be concentrated on a two-dimensional surface in 3-space), but the $3ţimes 3$ covariance matrix would not be singular (that is, it would have an inverse), so the invertibility of the covariance matrix does not guarantee a non-singular distribution.

A simple example in R:

n <- 10000
set.seed(7*11*13)
X <- rnorm(n)
Y <- rnorm(n)
f <- function(x, y) x + y^3
Z <- f(X, Y)
dat <- cbind(X, Y, Z)

C <- cov(dat)
C

X  0.998418216 -0.005227852  1.012374
Y -0.005227852  0.984617994  2.955338
Z  1.012373644  2.955338277 16.487973
> svd(C)
$d
[1] 17.0937095  1.0001744  0.3771255

$u
            [,1]          [,2]       [,3]
[1,] -0.06169015  0.9460834489 -0.3179944
[2,] -0.18008324 -0.3239229950 -0.9287863
[3,] -0.98171501 -0.0000315037  0.1903566

$v
            [,1]          [,2]       [,3]
[1,] -0.06169015  0.9460834489 -0.3179944
[2,] -0.18008324 -0.3239229950 -0.9287863
[3,] -0.98171501 -0.0000315037  0.1903566

As all the singular values are positive, the covariance matrix is not singular.