# Linear transformation of multivariate normals resulting in a singular covariance matrix

Given a vector x of $n$ independent standard normal RVs, and an $m\times n$ matrix A, I was expecting that the linear transformation y=Ax would result in a multivariate normal with a covariance matrix AA$^T$. However, with $m>n$ the resulting $m \times m$ matrix AA$^T$ is not of full rank (if I understand it correctly) and hence is not invertible.

This seems to indicate that there is some "redundancy" in the system of equations y=Ax and that the same amount of information could, somehow, be represented by $n \times n$ matrix A$^*$ instead. Is this correct? And if yes, how does one construct this reduced matrix A$^*$ and then link it with the original system?

I was thinking of maybe performing a principal component analysis on AA$^T$ and then using the column vectors (eigenvectors) with non-zero eigenvalues, but this would still be an $m \times k$ matrix where $k$ is the number of nonzero eigenvalues, so that doesn't seem to be a correct approach. Or maybe I am not applying it in a right way. Should I be taking a matrix made from "non-zero" eigenvalues and applying it to A instead?

The background is the following. I have a system which is driven by, say, 4 factors: 1 common and 3 specific. That means that all $y$'s depend on the same common factor, but only those $y$'s that belong to group $i$ depend on the $i$-th specific factor. So, all $y$'s are correlated via the common risk factor and additionally $y$'s belonging to category $i$ are further correlated between them via the $i$-th specific factor. (In reality, I have several categories of common and corresponding specific factors: 1st common has 4 specific, and 2nd common has 8 specific, etc.) There are other properties of $y$'s which do not depend on the factors but which are heterogeneous across $y$'s. So, that's why $m>n$.

I need to simulate the distribution of y to estimate its quantile, so I was looking to implement importance sampling to reduce variance of the estimate. Initialy, I was thinking of applying "exponential twisting" change of measure to y (which in this case amounts to changing the mean from $\mathbf{0}$ to $\mathbf{c}$) but this requires taking an inverse of the covariance matrix, which can't be done in this case.

I then thought that maybe I can apply the twisting to x instead by changing their mean from $\mathbf{0}$ to $\mathbf{b} = (\mathbf{A}^T\mathbf{A})^{-1}\mathbf{A}^T\mathbf{c}$ which is a solution to $\mathbf{c} = \mathbf{Ab}$. However, I now have another problem that $\mathbf{A}$ is not even full column rank, i.e. $rank(\mathbf{A})<n$, so it doesn't have even the left inverse and I can't calculate $\mathbf{b}$ as above.. This is because the columns of sensitivities to the specific factors can be combined linearly to give the column of sensitivities to the common factor. For example, say $m=6$ and $n=5$ factors: 1st common without any specific, and 2nd common with 3 specific; then $\mathbf{A}=$

\begin{matrix} b_{11} & b_{12} & a_{12} & 0 & 0\\ b_{21} & b_{12} & a_{12} & 0 & 0\\ b_{31} & b_{22} & 0 & a_{22} & 0\\ b_{11} & b_{22} & 0 & a_{22} & 0\\ b_{21} & b_{32} & 0 & 0 & a_{32}\\ b_{31} & b_{32} & 0 & 0 & a_{32} \end{matrix}

where $b_{ij}$ and $a_{ij}$ are sensitivities to the $j$-th common and specific factors of $y$ belonging to category $i$ of $j$.

So, the covariance between between $y$'s in the sage category of the 2nd common factor are, for example, $Cov(y_{1},y_{2}) = b_{11}b_{21} + b_{12}^2 + a_{12}^2$, while for those from different categories we have $Cov(y_{1},y_{3}) = b_{11}b_{31} + b_{12}b_{22}$.

If we also add specific risk factors for the 1st common factor, then A becomes

\begin{matrix} b_{11} & a_{11} & 0 & 0 & b_{12} & a_{12} & 0 & 0\\ b_{21} & 0 & a_{21} & 0 & b_{12} & a_{12} & 0 & 0\\ b_{31} & 0 & 0 & a_{31} & b_{22} & 0 & a_{22} & 0\\ b_{11} & a_{11} & 0 & 0 & b_{22} & 0 & a_{22} & 0\\ b_{21} & 0 & a_{21} & 0 & b_{32} & 0 & 0 & a_{32}\\ b_{31} & 0 & 0 & a_{31} & b_{32} & 0 & 0 & a_{32} \end{matrix}

• What do you want to do with $y$? Why is $m > n$? Until you answer that, I don't think we can advise you what to do. Cov(y), which = $AA^T$, is singular because you embedded an object (x) having support in n dimensions, in a higher (m) dimensional space, in which it still only has support in n dimensions. Commented May 10, 2018 at 19:26

You are correct in your assessment that $\mathbf y = \mathbf{Ax}$ is multivariate normal with covariance matrix $\mathbf{AA}^T$ when $\mathbf x$ is a vector of $n$ independent standard normal random variables, and that when $m > n$, $\mathbf{AA}^T$ is not invertible. But all that that means is that the $m$ random variables comprising $\mathbf y$ do not enjoy an $m$-variate joint density function; they still do have multivariate normal distribution and have all the properties that such distributions imply.
There really is no need to construct a reduced matrix $\mathbf A^*$ to represent the $m$ random variables in $\mathbf y$ because any question if interest that can be asked about $\mathbf y$ can already be expressed (and answered!) in terms of $\mathbf x$, which is already a collection of $n$ independent normal random variables.
• Thank you for your reply. What if, say, I didn't know that $\mathbf{y} = \mathbf{Ax}$ and was just "given" a covariance matrix for $m$ variables which was singular. If I understand it correctly, this means that there is a linear dependent between some variables. How would one proceed to reduce this system (represented by this covariance matrix) into one where there is no linear dependence and hence invertible covariance matrix? Commented May 11, 2018 at 8:27