0
$\begingroup$

This is from Theodoridis' Machine Learning, exercise 3.16.

Suppose $\mathbf{x}$ is a vector of jointly normal random variables with covariance matrix $\boldsymbol\Sigma_x$. Let $$\mathbf{y} = \boldsymbol\Theta\mathbf{x}+\boldsymbol\eta$$ where $\boldsymbol\Theta$ is a $k \times l$ matrix of [I assume, known or fixed] parameters and $\boldsymbol\eta$ is normal with mean $\mathbf{0}$ and covariance matrix $\boldsymbol\Sigma_\eta$, independent of $\mathbf{x}$. Show that $\mathbf{y}$ and $\mathbf{x}$ are jointly Gaussian with covariance matrix $$\boldsymbol\Sigma = \begin{bmatrix} \boldsymbol\Theta\boldsymbol\Sigma_x\boldsymbol\Theta^{T}+\boldsymbol\Sigma_\eta & \boldsymbol\Theta\boldsymbol\Sigma_x \\ \boldsymbol\Sigma_x\boldsymbol\Theta^{T} & \boldsymbol\Sigma_x \end{bmatrix}\text{.}$$

It is clear that $\mathbf{y} \sim \mathcal{N}_k(\boldsymbol\Theta\boldsymbol\mu_x,\boldsymbol\Theta\boldsymbol\Sigma_x\boldsymbol\Theta^{T}+\boldsymbol\Sigma_\eta)$. This explains the top-left diagonal of $\boldsymbol\Sigma$.

The $\boldsymbol\Theta\boldsymbol\Sigma_x$, as well as the $\boldsymbol\Sigma_x\boldsymbol\Theta^{T}$, are easy to explain as well.

However, I'm not sure how to show that $\mathbf{y}$ and $\mathbf{x}$ are jointly Gaussian. In particular, I considered looking at the distribution of $\mathbf{y} \mid \mathbf{x}$, but using this assumes bivariate normality of $(\mathbf{x}, \mathbf{y})$ to start.

$\endgroup$
1
$\begingroup$

It's simply an application of linear algebra to start working on with $x$ and $y$ jointly (no need to study the conditional distribution). Since we are interested in looking for the joint distribution of $[y', x']'$, it is natural by condition to express it as \begin{align} \begin{bmatrix} y \\ x \end{bmatrix} = \begin{bmatrix} \Theta x + \eta \\ x \end{bmatrix} = \begin{bmatrix} \Theta & I \\ I & 0 \end{bmatrix} \begin{bmatrix} x \\ \eta \end{bmatrix}. \tag{1} \end{align}

Since $x$ and $\eta$ are independent normal vectors respectively, it follows that \begin{align} \begin{bmatrix} x \\ \eta \end{bmatrix} \sim N\left(\begin{bmatrix} \mu_x \\ 0 \end{bmatrix}, \begin{bmatrix} \Sigma_x & 0 \\ 0 & \Sigma_\eta \end{bmatrix}\right). \end{align}

Using the well-known affine transformation property of normal random vector, in view of $(1)$, $[y', x']'$ has multivariate normal distribution with the mean vector $$\begin{bmatrix} \Theta & I \\ I & 0 \end{bmatrix} \begin{bmatrix} \mu_x \\ 0 \end{bmatrix} = \begin{bmatrix} \Theta \mu_x \\ \mu_x \end{bmatrix}$$ and the covariance matrix $$\begin{bmatrix} \Theta & I \\ I & 0 \end{bmatrix} \begin{bmatrix} \Sigma_x & 0 \\ 0 & \Sigma_\eta \end{bmatrix} \begin{bmatrix} \Theta & I \\ I & 0 \end{bmatrix}' = \begin{bmatrix} \Theta\Sigma_x & \Sigma_\eta \\ \Sigma_x & 0 \end{bmatrix} \begin{bmatrix} \Theta' & I \\ I & 0 \end{bmatrix} = \begin{bmatrix} \Theta\Sigma_x\Theta' + \Sigma_\eta & \Theta\Sigma_x \\ \Sigma_x\Theta' & \Sigma_x \end{bmatrix}.$$ This completes the proof.

$\endgroup$
  • $\begingroup$ On one of my attempts to write down this problem, I wrote the equation $(1)$ but didn't know where to go from there... and this makes complete sense. Thanks! $\endgroup$ – Clarinetist Jan 2 '18 at 4:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.