Show that $(\mathbf{x}, \boldsymbol\Theta\mathbf{x}+\boldsymbol\eta)$ is jointly normal

Question

This is from Theodoridis' Machine Learning, exercise 3.16.

Suppose $\mathbf{x}$ is a vector of jointly normal random variables with covariance matrix $\boldsymbol\Sigma_x$. Let $$\mathbf{y} = \boldsymbol\Theta\mathbf{x}+\boldsymbol\eta$$ where $\boldsymbol\Theta$ is a $k \times l$ matrix of [I assume, known or fixed] parameters and $\boldsymbol\eta$ is normal with mean $\mathbf{0}$ and covariance matrix $\boldsymbol\Sigma_\eta$, independent of $\mathbf{x}$. Show that $\mathbf{y}$ and $\mathbf{x}$ are jointly Gaussian with covariance matrix $$\boldsymbol\Sigma = \begin{bmatrix} \boldsymbol\Theta\boldsymbol\Sigma_x\boldsymbol\Theta^{T}+\boldsymbol\Sigma_\eta & \boldsymbol\Theta\boldsymbol\Sigma_x \\ \boldsymbol\Sigma_x\boldsymbol\Theta^{T} & \boldsymbol\Sigma_x \end{bmatrix}\text{.}$$

It is clear that $\mathbf{y} \sim \mathcal{N}_k(\boldsymbol\Theta\boldsymbol\mu_x,\boldsymbol\Theta\boldsymbol\Sigma_x\boldsymbol\Theta^{T}+\boldsymbol\Sigma_\eta)$. This explains the top-left diagonal of $\boldsymbol\Sigma$.

The $\boldsymbol\Theta\boldsymbol\Sigma_x$, as well as the $\boldsymbol\Sigma_x\boldsymbol\Theta^{T}$, are easy to explain as well.

However, I'm not sure how to show that $\mathbf{y}$ and $\mathbf{x}$ are jointly Gaussian. In particular, I considered looking at the distribution of $\mathbf{y} \mid \mathbf{x}$, but using this assumes bivariate normality of $(\mathbf{x}, \mathbf{y})$ to start.

Answer 1

It's simply an application of linear algebra to start working on with $x$ and $y$ jointly (no need to study the conditional distribution). Since we are interested in looking for the joint distribution of $[y', x']'$, it is natural by condition to express it as \begin{align} \begin{bmatrix} y \\ x \end{bmatrix} = \begin{bmatrix} \Theta x + \eta \\ x \end{bmatrix} = \begin{bmatrix} \Theta & I \\ I & 0 \end{bmatrix} \begin{bmatrix} x \\ \eta \end{bmatrix}. \tag{1} \end{align}

Since $x$ and $\eta$ are independent normal vectors respectively, it follows that \begin{align} \begin{bmatrix} x \\ \eta \end{bmatrix} \sim N\left(\begin{bmatrix} \mu_x \\ 0 \end{bmatrix}, \begin{bmatrix} \Sigma_x & 0 \\ 0 & \Sigma_\eta \end{bmatrix}\right). \end{align}

Using the well-known affine transformation property of normal random vector, in view of $(1)$, $[y', x']'$ has multivariate normal distribution with the mean vector $$\begin{bmatrix} \Theta & I \\ I & 0 \end{bmatrix} \begin{bmatrix} \mu_x \\ 0 \end{bmatrix} = \begin{bmatrix} \Theta \mu_x \\ \mu_x \end{bmatrix}$$ and the covariance matrix $$\begin{bmatrix} \Theta & I \\ I & 0 \end{bmatrix} \begin{bmatrix} \Sigma_x & 0 \\ 0 & \Sigma_\eta \end{bmatrix} \begin{bmatrix} \Theta & I \\ I & 0 \end{bmatrix}' = \begin{bmatrix} \Theta\Sigma_x & \Sigma_\eta \\ \Sigma_x & 0 \end{bmatrix} \begin{bmatrix} \Theta' & I \\ I & 0 \end{bmatrix} = \begin{bmatrix} \Theta\Sigma_x\Theta' + \Sigma_\eta & \Theta\Sigma_x \\ \Sigma_x\Theta' & \Sigma_x \end{bmatrix}.$$ This completes the proof.