Paper: A Unifying Review of Linear Gaussian Models by Roweis & Ghahramani

The generative model is the typical state space model written as \begin{align} \text{state transition equation: }{\bf x}_t &= {\bf A} {\bf x}_{t-1} + {\bf w}_t && {\bf w}_t \sim \mathcal{N} \left( {\bf 0}, {\bf Q} \right) \\ \text{observation equation: }{\bf y}_t &= {\bf C} {\bf x}_t + {\bf v}_t && {\bf v}_t \sim \mathcal{N} \left( {\bf 0}, {\bf R} \right) \end{align} where ${\bf A}$ is the $k \times k$ state transition matrix and ${\bf C}$ is the $p \times k$ observation matrix.

In the paper on. page 2, the authors write

Notice that there is degeneracy in the model: all of the structure in the matrix $\bf Q$ can be moved into the matrices $\bf A$ and $\bf C$. This means we can without loss generality work with models in which $\bf Q$ is the identity matrix.

There is a footnote associated with the passage and it reads

In particular, since is it a covariance matrix, $\bf Q$ is symmetric positive semi-definite and thus can be diagonalized to the form $\bf E \Lambda E^{\top}$ (where $\bf E$ is the rotation matrix of eigenvectors and ${\bf \Lambda}$ is a diagonal matrix of eigenvalues). Thus for any model in which $\bf Q$ is not the identity matrix, we can generate an exactly equivalent model using a new state vector ${\bf x}' = {\bf \Lambda}^{-1/2} {\bf E}^{\top} {\bf x}$ with ${\bf A}' = ( {\bf \Lambda}^{-1/2} {\bf E}^{\top} ) \, {\bf A} \, ( {\bf E} {\bf \Lambda}^{1/2} )$ and ${\bf C}' = {\bf C} ( {\bf E} {\bf \Lambda}^{1/2})$ such that the new covariance of ${\bf x}'$ is the identity matrix: ${\bf Q}' = {\bf I}$.

Question: How is the new covariance of $\bf x'$ equal to the identity matrix?

\begin{align} \text{Var} ({\bf x}') = {\bf \Lambda}^{-1/2} {\bf E}^{\top} \text{Var} ({\bf x}) {\bf E} {\bf \Lambda}^{-1/2} \end{align} equals the identity matrix if $\text{Var} ({\bf x})$ is the identity matrix. What in the model tells use that this is the case?

Update Mar 27: I believe the authors are referring to the conditional covariance in the footnote.

From the generative model we can see that the conditional distribution of ${\bf x}_t \, \vert \, {\bf x}_{t-1}$ is Gaussian with conditional mean ${\bf A} {\bf x}_{t-1}$ and conditional covariance ${\bf Q}$.

Using the transformed state vector, the new generative model is now \begin{align} \text{new state transition equation: }{\bf x}'_t &= {\bf A}' {\bf x}'_{t-1} + {\bf w}'_t && {\bf w}'_t \sim \mathcal{N} \left( {\bf 0}, {\bf I} \right) \\ \text{new observation equation: }{\bf y}_t &= {\bf C}' {\bf x}'_t + {\bf v}_t && {\bf v}_t \sim \mathcal{N} \left( {\bf 0}, {\bf R} \right). \end{align}

Thus, the conditional distribution of ${\bf x}'_t \, \vert \, {\bf x}'_{t-1}$ is Gaussian with conditional mean ${\bf A}' {\bf x}'_{t-1}$ and conditional covariance ${\bf I}$.

  • $\begingroup$ yep +1 for the edit--multiply both sides of the state equation by ${\bf \Lambda}^{-1/2} {\bf E}^{\top}$ and rewrite the $\mathbf{x}$s as $\mathbf{x}'$s $\endgroup$ – Taylor Mar 28 at 2:01

I think you have it backwards. $\bf Q$, not $\bf I$ is the variance of $\bf x$.

The intended result can become clearer from some quick matrix multiplications to rearrange your equation.

$$ \begin{align} \text{Var} ({\bf x}') &= {\bf \Lambda}^{-1/2} {\bf E}^{\top} \text{Var} ({\bf x}) {\bf E} {\bf \Lambda}^{-1/2} \\ {\bf \Lambda}^{1/2} \text{Var} ({\bf x}') &= {\bf E}^{\top} \text{Var} ({\bf x}) {\bf E} {\bf \Lambda}^{-1/2} \\ {\bf E}{\bf \Lambda}^{1/2} \text{Var} ({\bf x}') &= \text{Var} ({\bf x}) {\bf E} {\bf \Lambda}^{-1/2} \\ &\vdots \\ {\bf E}{\bf \Lambda}^{1/2} \text{Var} ({\bf x}') {\bf \Lambda}^{1/2} {\bf E}^{\top} &= \text{Var} ({\bf x}) \\ \end{align} $$

We know from the given noise process that $\text{Var} ({\bf x}) = {\bf Q}$. Now if we wager that $\text{Var} ({\bf x'}) = {\bf I}$, we see that the left-hand side of the equation becomes our eigendecomposition of ${\bf Q}$, so the two sides are equal. That is, the variance of $\bf x'$ is $\bf I$ when the variance of $\bf x$ is $\bf Q$.

Alternatively, you could go the direct route.

$$ \begin{align} \text{Var} ({\bf x}') &= {\bf \Lambda}^{-1/2} {\bf E}^{\top} \text{Var} ({\bf x}) {\bf E} {\bf \Lambda}^{-1/2} \\ \text{Var} ({\bf x}') &= {\bf \Lambda}^{-1/2} {\bf E}^{\top} {\bf Q} {\bf E} {\bf \Lambda}^{-1/2} \\ \text{Var} ({\bf x}') &= {\bf \Lambda}^{-1/2} {\bf E}^{\top} {\bf E} {\bf \Lambda} {\bf E}^{\top} {\bf E} {\bf \Lambda}^{-1/2} \\ \text{Var} ({\bf x}') &= {\bf \Lambda}^{-1/2} {\bf \Lambda} {\bf \Lambda}^{-1/2} \\ \text{Var} ({\bf x}') &= {\bf I} \\ \end{align} $$

  • $\begingroup$ I am having a hard time seeing how $\text{Var} ({\bf x}) = {\bf Q}$. I know that $\text{Var} ({\bf w}) = {\bf Q}$ by the way the generative model is defined. How does it follow that they share the same variance? $\endgroup$ – SOULed_Outt Mar 27 at 2:26
  • $\begingroup$ The conditional variance $\text{Var} ({\bf x}_t \, \vert \, {\bf x}_{t-1} ) = {\bf Q}$. I am certain that the marginal variance $\text{Var} ({\bf x}_t ) = {\bf Q} + \text{something}$ and is only equal to the conditional variance if you make additional assumptions about the model. $\endgroup$ – SOULed_Outt Mar 27 at 18:23
  • 1
    $\begingroup$ @SOULed_Outt the marginal variance doesn't always exist....unless you have a stationary chain, in which case it usually isn't equal to the conditional covariance matrix $\endgroup$ – Taylor Mar 28 at 2:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.