Multivariate normal draws with specified correlations, standard deviations -- are they not "independent"?

Question

I thought it'd be possible to use any combination of correlations and standard deviations to generate correlated standard normal draws.

The intent is to get around checking whether the variance-covariance matrix $\Sigma$ is positive-definite.

In the three-variable case, my idea is, given a vector of standard deviations $(\sigma_X, \sigma_Y, \sigma_Z)$ and a triple of correlations, $(\rho_{XY}, \rho_{XZ}, \rho_{YZ})$, we construct $\Sigma$ as

$$\Sigma = \begin{bmatrix} \sigma_X & 0 & 0 \\ 0 & \sigma_Y & 0 \\ 0 & 0 & \sigma_Z \end{bmatrix} \begin{bmatrix} 1 & \rho_{XY} & \rho_{XZ} \\ \rho_{XY} & 1 & \rho_{YZ} \\ \rho_{XZ} & \rho_{YZ} & 1 \end{bmatrix} \begin{bmatrix} \sigma_X & 0 & 0 \\ 0 & \sigma_Y & 0 \\ 0 & 0 & \sigma_Z \end{bmatrix} = \begin{bmatrix} \sigma_X^2 & \sigma_{XY} & \sigma_{XZ} \\ \sigma_{XY} & \sigma_Y^2 & \sigma_{YZ} \\ \sigma_{XZ} & \sigma_{YZ} & \sigma_Z^2 \end{bmatrix} $$

With $\Sigma$ in hand, we draw standard normal variates $Z$ and correlate them with the transformation

$$ F = L\Sigma $$

Where $L$ is such that $LL^{T} = \Sigma$ (e.g., Cholesky decomposition). And voila, $F \sim N(0, \Sigma)$!

I thought we'd be able to specify whatever (valid) values of $\sigma_k$ and $\rho_{ij}$ we'd like, but in coding this up I seem to have created an invalid $\Sigma$:

sds = c(.05, .05, .05)
corr = c(0, .385, .97)

#generate center matrix
corr_mat = diag(3)
corr_mat[lower.tri(corr_mat)] = corr_mat[upper.tri(corr_mat)] = corr

#generate sigma
sigma = diag(sds) %*% corr_mat %*% diag(sds)

#get L
chol(sigma)

Error in chol.default(sigma) : the leading minor of order 3 is not positive definite

How can $\Sigma$ not be positive definite? Is there some restriction on the relative magnitude of standard deviations and correlations? I thought the answer is no...

$\Sigma$ is definitely not positive definite:

eigen(sigma)$values
# [1]  0.0051090288  0.0025000000 -0.0001090288

I see the negative eigenvalue is very small... is this possibly just a numerical problem?

Answer 1

The issue is that your correlation matrix is not positive definite. You can get this in lot's of ways. One possible way occurs when $cor(X,Y) = .9$, $cor(X,Z) = .9$, $cor(Y,Z) = 0$ this isn't possible for normal variables, and will give you a non-positive definite matrix with eigenvalues (2.2727922, 1.0000000, -0.2727922).

We can show this with a bit more rigor in the case of the random variables $X$, $Y$, and $Z$ distributed $\mathcal{N}(0,1)$. For all three variables, we can write each as the sum of two independent random variables distributed $\mathcal{N}(0,1)$ and multiplied by scalars such that $X=\alpha_1X_1 + \alpha_2X_2$ and $var(X) = \alpha_1^2 + \alpha_2^2 =1$ :

In the case of $X$ and $Y$, where $\mathbb{E}(X_1Y_1)=1$, $\mathbb{E}(X_1Y_2)=0$, and $\mathbb{E}(X_2Y) = 0$ then $\mathbb{E}(XY)=\mathbb{E}((\alpha_1X_1 + \alpha_2X_2)(\beta_1Y_1 + \beta_2Y_2))=\alpha_1\beta_1$.

In the case of $X$ and $Z$, where $\mathbb{E}(X_1Z_1)=1$, $\mathbb{E}(X_1Z_2)=0$, and $\mathbb{E}(X_2Z) = 0$ then $\mathbb{E}(XZ)=\mathbb{E}((\alpha_3X_3 + \alpha_4X_4)(\gamma_1Z_1 + \gamma_2Z_2))=\alpha_3\gamma_1$.

The correlation between $X_1$ and $X_3$ can be zero, if $\alpha_1^2 + \alpha_3^2 < 1$, but if $\mathbb{E}(X,Y)$ and $\mathbb{E}(X,Z)$ are large this may not be possible.

To find values that work, you need to ensure the determinants of the principal minors are positive. To do this use the $2 \times 2$ and $3 \times 3$ formulas for the determinant, plug in the values you know (the diagonal), and check that the resulting value:

The $2\times 2$ Determinant

$\left| \begin{array}{cc} 1 & a \\ a & 1 \end{array} \right| = 1 - a^2 > 0$

The $3\times 3$ Determinant

$\left| \begin{array}{ccc} 1 & a & b \\ a & 1 & c \\ b & c & 1 \end{array} \right| = 1 + 2abc - b^2 - a^2 - c^2 > 0$