Return to Answer

The following post contains an outline of a proof, just to give the main ideas and get you started.

Let $z = (Z_1, Z_2)$ be two independent Gaussian random variables and let $x = (X_1, X_2)$ be $$ x = \begin{pmatrix} X_1 \\ X_2 \end{pmatrix} = \begin{pmatrix} \alpha_{11} Z_1 + \alpha_{12} Z_2\\ \alpha_{21} Z_1 + \alpha_{22} Z_2 \end{pmatrix} = \begin{pmatrix} \alpha_{11} & \alpha_{12}\\ \alpha_{21} & \alpha_{22} \end{pmatrix} \begin{pmatrix} Z_1 \\ Z_2 \end{pmatrix} = A z. $$

Each $X_i \sim N(\mu_i, \sigma_i^2)$, but as they are both linear combinations of the same independent r.vs, they are jointly dependent.

Definition A pair of r.vs $x = (X_1, X_2)$ are said to be bivariate normally distributed iff it can be written as a linear combination $x = Az$ of independent normal r.vs $z = (Z_1, Z_2)$.

Lemma If $ x = (X_1, X_2)$ is a bivariate Gaussian, then any other linear combination of them is again a normal random variable.

Proof. Trivial, skipped to not offend anyone.

Property If $X_1, X_2$ are uncorrelated, then they are independent and vice-versa.

Distribution of $X_1 | X_2$

Assume $X_1, X_2$ are the same Gaussian r.vs as before but let's suppose they have positive variance and zero mean for simplicity.

If $\mathbf S$ is the subspace spanned by $X_2$, let $ X_1^{\mathbf S} = \frac{\rho \sigma_{X_1}}{\sigma_{X_2}} X_2 $ and $ X_1^{\mathbf S^\perp} = X_1 - X_1^{\mathbf S} $.

$X_1$ and $X_2$ are linear combinations of Gaussians, so they are Gaussians too. But this also holds for $ X_2, X_1^{\mathbf S^\perp}$ which are also uncorrelated (prove it) and independent.

The decomposition $$ X_1 = X_1^{\mathbf S} + X_1^{\mathbf S^\perp} $$ holds with $\mathbf{E}[X_1 | X_2] = \frac{\rho \sigma_{X_1}}{\sigma_{X_2}} X_2 = X_1^{\mathbf S}$

$$ \begin{split} \mathbf{V}[X_1 | X_2] &= \mathbf{V}[X_1^{\mathbf S^\perp}] \\ &= \mathbf{E} \left[ X_1 - \frac{\rho \sigma_{X_1}}{\sigma_{X_2}} X_2 \right]^2 \\ &= (1 - \rho)^2 \sigma^2_{X_1}. \end{split} $$

Then $$ X_1 | X_2 \sim N\left( X_1^{\mathbf S}, (1 - \rho)^2 \sigma^2_{X_1} \right).$$

Two univariate Gaussian random variables $X, Y$ are jointly Gaussian if the conditionals $X | Y$ and $Y|X$ are Gaussian too.

The following post contains an outline of a proof, just to give the main ideas and get you started.

Let $z = (Z_1, Z_2)$ be two independent Gaussian random variables and let $x = (X_1, X_2)$ be $$ x = \begin{pmatrix} X_1 \\ X_2 \end{pmatrix} = \begin{pmatrix} \alpha_{11} Z_1 + \alpha_{12} Z_2\\ \alpha_{21} Z_1 + \alpha_{22} Z_2 \end{pmatrix} = \begin{pmatrix} \alpha_{11} & \alpha_{12}\\ \alpha_{21} & \alpha_{22} \end{pmatrix} \begin{pmatrix} Z_1 \\ Z_2 \end{pmatrix} = A z. $$

Each $X_i \sim N(\mu_i, \sigma_i^2)$, but as they are both linear combinations of the same independent r.vs, they are jointly dependent.

Definition A pair of r.vs $x = (X_1, X_2)$ are said to be bivariate normally distributed iff it can be written as a linear combination $x = Az$ of independent normal r.vs $z = (Z_1, Z_2)$.

Lemma If $ x = (X_1, X_2)$ is a bivariate Gaussian, then any other linear combination of them is again a normal random variable.

Proof. Trivial, skipped to not offend anyone.

Property If $X_1, X_2$ are uncorrelated, then they are independent and vice-versa.

Distribution of $X_1 | X_2$

Assume $X_1, X_2$ are the same Gaussian r.vs as before but let's suppose they have positive variance and zero mean for simplicity.

If $\mathbf S$ is the subspace spanned by $X_2$, let $ X_1^{\mathbf S} = \frac{\rho \sigma_{X_1}}{\sigma_{X_2}} X_2 $ and $ X_1^{\mathbf S^\perp} = X_1 - X_1^{\mathbf S} $.

$X_1$ and $X_2$ are linear combinations of $z$, so $ X_2, X_1^{\mathbf S^\perp}$ are too. They are jointly Gaussian, uncorrelated (prove it) and independent.

The decomposition $$ X_1 = X_1^{\mathbf S} + X_1^{\mathbf S^\perp} $$ holds with $\mathbf{E}[X_1 | X_2] = \frac{\rho \sigma_{X_1}}{\sigma_{X_2}} X_2 = X_1^{\mathbf S}$

$$ \begin{split} \mathbf{V}[X_1 | X_2] &= \mathbf{V}[X_1^{\mathbf S^\perp}] \\ &= \mathbf{E} \left[ X_1 - \frac{\rho \sigma_{X_1}}{\sigma_{X_2}} X_2 \right]^2 \\ &= (1 - \rho)^2 \sigma^2_{X_1}. \end{split} $$

Then $$ X_1 | X_2 \sim N\left( X_1^{\mathbf S}, (1 - \rho)^2 \sigma^2_{X_1} \right).$$

Two univariate Gaussian random variables $X, Y$ are jointly Gaussian if the conditionals $X | Y$ and $Y|X$ are Gaussian too.

Without going into too complex explanations involving copulas, which are perfectly fine though, just think about the product rule for two random variables $X$ and $Y$ both having Gaussian distributions: $$ f_{X,Y}(x,y) = f_{X}(x) f_{Y \mid X}(y \mid x) = f_{Y}(y) f_{X \mid Y}(x \mid y). $$ The marginals are normal pdfs, then in order to have a joint normal you need the conditional $ f_{Y \mid X}$ to be normal too!

Best,

Ancillary

The following post contains an outline of a proof, just to give the main ideas and get you started.

Let $z = (Z_1, Z_2)$ be two independent Gaussian random variables and let $x = (X_1, X_2)$ be $$ x = \begin{pmatrix} X_1 \\ X_2 \end{pmatrix} = \begin{pmatrix} \alpha_{11} Z_1 + \alpha_{12} Z_2\\ \alpha_{21} Z_1 + \alpha_{22} Z_2 \end{pmatrix} = \begin{pmatrix} \alpha_{11} & \alpha_{12}\\ \alpha_{21} & \alpha_{22} \end{pmatrix} \begin{pmatrix} Z_1 \\ Z_2 \end{pmatrix} = A z. $$

Each $X_i \sim N(\mu_i, \sigma_i^2)$, but as they are both linear combinations of the same independent r.vs, they are jointly dependent.

Definition A pair of r.vs $x = (X_1, X_2)$ are said to be bivariate normally distributed iff it can be written as a linear combination $x = Az$ of independent normal r.vs $z = (Z_1, Z_2)$.

Lemma If $ x = (X_1, X_2)$ is a bivariate Gaussian, then any other linear combination of them is again a normal random variable.

Proof. Trivial, skipped to not offend anyone.

Property If $X_1, X_2$ are uncorrelated, then they are independent and vice-versa.

Distribution of $X_1 | X_2$

Assume $X_1, X_2$ are the same Gaussian r.vs as before but let's suppose they have positive variance and zero mean for simplicity.

If $\mathbf S$ is the subspace spanned by $X_2$, let $ X_1^{\mathbf S} = \frac{\rho \sigma_{X_1}}{\sigma_{X_2}} X_2 $ and $ X_1^{\mathbf S^\perp} = X_1 - X_1^{\mathbf S} $.

$X_1$ and $X_2$ are linear combinations of Gaussians, so they are Gaussians too. But this also holds for $ X_2, X_1^{\mathbf S^\perp}$ which are also uncorrelated (prove it) and independent.

The decomposition $$ X_1 = X_1^{\mathbf S} + X_1^{\mathbf S^\perp} $$ holds with $\mathbf{E}[X_1 | X_2] = \frac{\rho \sigma_{X_1}}{\sigma_{X_2}} X_2 = X_1^{\mathbf S}$

$$ \begin{split} \mathbf{V}[X_1 | X_2] &= \mathbf{V}[X_1^{\mathbf S^\perp}] \\ &= \mathbf{E} \left[ X_1 - \frac{\rho \sigma_{X_1}}{\sigma_{X_2}} X_2 \right]^2 \\ &= (1 - \rho)^2 \sigma^2_{X_1}. \end{split} $$

Then $$ X_1 | X_2 \sim N\left( X_1^{\mathbf S}, (1 - \rho)^2 \sigma^2_{X_1} \right).$$

Two univariate Gaussian random variables $X, Y$ are jointly Gaussian if the conditionals $X | Y$ and $Y|X$ are Gaussian too.