Return to Question

Given a vector x of $n$ independent standard normal RVs, and an $m\times n$ matrix A, I was expecting that the linear transformation y=Ax would result in a multivariate normal with a covariance matrix AA$^T$. However, with $m>n$ the resulting $m \times m$ matrix AA$^T$ is not of full rank (if I understand it correctly) and hence is not invertible.

This seems to indicate that there is some "redundancy" in the system of equations y=Ax and that the same amount of information could, somehow, be represented by $n \times n$ matrix A$^*$ instead. Is this correct? And if yes, how does one construct this reduced matrix A$^*$ and then link it with the original system?

I was thinking of maybe performing a principal component analysis on AA$^T$ and then using the column vectors (eigenvectors) with non-zero eigenvalues, but this would still be an $m \times k$ matrix where $k$ is the number of nonzero eigenvalues, so that doesn't seem to be a correct approach. Or maybe I am not applying it in a right way. Should I be taking a matrix made from "non-zero" eigenvalues and applying it to A instead?

Add 1

The background is the following. I have a system which is driven by, say, 4 factors: 1 common and 3 specific. That means that all $y$'s depend on the same common factor, but only those $y$'s that belong to group $i$ depend on the $i$-th specific factor. So, all $y$'s are correlated via the common risk factor and additionally $y$'s belonging to category $i$ are further correlated between them via the $i$-th specific factor. (In reality, I have several categories of common and corresponding specific factors: 1st common has 4 specific, and 2nd common has 8 specific, etc.) There are other properties of $y$'s which do not depend on the factors but which are heterogeneous across $y$'s. So, that's why $m>n$.

I need to simulate the distribution of y to estimate its quantile, so I was looking to implement importance sampling to reduce variance of the estimate. Initialy, I was thinking of applying "exponential twisting" change of measure to y (which in this case amounts to changing the mean from $\mathbf{0}$ to $\mathbf{c}$) but this requires taking an inverse of the covariance matrix, which can't be done in this case.

I then thought that maybe I can apply the twisting to x instead by changing their mean from $\mathbf{0}$ to $\mathbf{b} = (\mathbf{A}^T\mathbf{A})^{-1}\mathbf{A}^T\mathbf{c}$ which is a solution to $\mathbf{c} = \mathbf{Ab}$. However, I now have another problem that $\mathbf{A}$ is not even full column rank, i.e. $rank(\mathbf{A})<n$, so it doesn't have even the left inverse and I can't calculate $\mathbf{b}$ as above.. This is because the columns of sensitivities to the specific factors can be combined linearly to give the column of sensitivities to the common factor. For example, say $m=6$ and $n=5$ factors: 1st common without any specific, and 2nd common with 3 specific; then $\mathbf{A}=$

\begin{matrix} b_{11} & b_{12} & a_{12} & 0 & 0\\ b_{21} & b_{12} & a_{12} & 0 & 0\\ b_{31} & b_{22} & 0 & a_{22} & 0\\ b_{41} & b_{22} & 0 & a_{22} & 0\\ b_{51} & b_{32} & 0 & 0 & a_{32}\\ b_{61} & b_{32} & 0 & 0 & a_{32} \end{matrix}

where $b_{ij}$ and $a_{ij}$ are sensitivities to the $j$-th common and specific factors of $y$ belonging to category $i$ of $j$.

So, the covariance between between $y$'s in the sage category of the 2nd common factor are, for example, $Cov(y_{1},y_{2}) = b_{11}b_{21} + b_{12}^2 + a_{12}^2$, while for those from different categories we have $Cov(y_{1},y_{3}) = b_{11}b_{31} + b_{12}b_{22}$

Given a vector x of $n$ independent standard normal RVs, and an $m\times n$ matrix A, I was expecting that the linear transformation y=Ax would result in a multivariate normal with a covariance matrix AA$^T$. However, with $m>n$ the resulting $m \times m$ matrix AA$^T$ is not of full rank (if I understand it correctly) and hence is not invertible.

This seems to indicate that there is some "redundancy" in the system of equations y=Ax and that the same amount of information could, somehow, be represented by $n \times n$ matrix A$^*$ instead. Is this correct? And if yes, how does one construct this reduced matrix A$^*$ and then link it with the original system?

I was thinking of maybe performing a principal component analysis on AA$^T$ and then using the column vectors (eigenvectors) with non-zero eigenvalues, but this would still be an $m \times k$ matrix where $k$ is the number of nonzero eigenvalues, so that doesn't seem to be a correct approach. Or maybe I am not applying it in a right way. Should I be taking a matrix made from "non-zero" eigenvalues and applying it to A instead?

Add 1

The background is the following. I have a system which is driven by, say, 4 factors: 1 common and 3 specific. That means that all $y$'s depend on the same common factor, but only those $y$'s that belong to group $i$ depend on the $i$-th specific factor. So, all $y$'s are correlated via the common risk factor and additionally $y$'s belonging to category $i$ are further correlated between them via the $i$-th specific factor. (In reality, I have several categories of common and corresponding specific factors: 1st common has 4 specific, and 2nd common has 8 specific, etc.) There are other properties of $y$'s which do not depend on the factors but which are heterogeneous across $y$'s. So, that's why $m>n$.

I need to simulate the distribution of y to estimate its quantile, so I was looking to implement importance sampling to reduce variance of the estimate. Initialy, I was thinking of applying "exponential twisting" change of measure to y (which in this case amounts to changing the mean from $\mathbf{0}$ to $\mathbf{c}$) but this requires taking an inverse of the covariance matrix, which can't be done in this case.

I then thought that maybe I can apply the twisting to x instead by changing their mean from $\mathbf{0}$ to $\mathbf{b} = (\mathbf{A}^T\mathbf{A})^{-1}\mathbf{A}^T\mathbf{c}$ which is a solution to $\mathbf{c} = \mathbf{Ab}$. However, I now have another problem that $\mathbf{A}$ is not even full column rank, i.e. $rank(\mathbf{A})<n$, so it doesn't have even the left inverse and I can't calculate $\mathbf{b}$ as above.. This is because the columns of sensitivities to the specific factors can be combined linearly to give the column of sensitivities to the common factor. For example, say $m=6$ and $n=5$ factors: 1st common without any specific, and 2nd common with 3 specific; then $\mathbf{A}=$

\begin{matrix} b_{11} & b_{12} & a_{12} & 0 & 0\\ b_{21} & b_{12} & a_{12} & 0 & 0\\ b_{31} & b_{22} & 0 & a_{22} & 0\\ b_{11} & b_{22} & 0 & a_{22} & 0\\ b_{21} & b_{32} & 0 & 0 & a_{32}\\ b_{31} & b_{32} & 0 & 0 & a_{32} \end{matrix}

where $b_{ij}$ and $a_{ij}$ are sensitivities to the $j$-th common and specific factors of $y$ belonging to category $i$ of $j$.

So, the covariance between between $y$'s in the sage category of the 2nd common factor are, for example, $Cov(y_{1},y_{2}) = b_{11}b_{21} + b_{12}^2 + a_{12}^2$, while for those from different categories we have $Cov(y_{1},y_{3}) = b_{11}b_{31} + b_{12}b_{22}$.

If we also add specific risk factors for the 1st common factor, then A becomes

\begin{matrix} b_{11} & a_{11} & 0 & 0 & b_{12} & a_{12} & 0 & 0\\ b_{21} & 0 & a_{21} & 0 & b_{12} & a_{12} & 0 & 0\\ b_{31} & 0 & 0 & a_{31} & b_{22} & 0 & a_{22} & 0\\ b_{11} & a_{11} & 0 & 0 & b_{22} & 0 & a_{22} & 0\\ b_{21} & 0 & a_{21} & 0 & b_{32} & 0 & 0 & a_{32}\\ b_{31} & 0 & 0 & a_{31} & b_{32} & 0 & 0 & a_{32} \end{matrix}

Given a vector x of $n$ independent standard normal RVs, and an $m\times n$ matrix A, I was expecting that the linear transformation y=Ax would result in a multivariate normal with a covariance matrix AA$^T$. However, with $m>n$ the resulting $m \times m$ matrix AA$^T$ is not of full rank (if I understand it correctly) and hence is not invertible.

This seems to indicate that there is some "redundancy" in the system of equations y=Ax and that the same amount of information could, somehow, be represented by $n \times n$ matrix A$^*$ instead. Is this correct? And if yes, how does one construct this reduced matrix A$^*$ and then link it with the original system?

I was thinking of maybe performing a principal component analysis on AA$^T$ and then using the column vectors (eigenvectors) with non-zero eigenvalues, but this would still be an $m \times k$ matrix where $k$ is the number of nonzero eigenvalues, so that doesn't seem to be a correct approach. Or maybe I am not applying it in a right way. Should I be taking a matrix made from "non-zero" eigenvalues and applying it to A instead?

Add 1

The background is the following. I have a system which is driven by, say, 4 factors: 1 common and 3 specific. That means that all $y$'s depend on the same common factor, but only those $y$'s that belong to group $i$ depend on the $i$-th specific factor. So, all $y$'s are correlated via the common risk factor and additionally $y$'s belonging to category $i$ are further correlated between them via the $i$-th specific factor. (In reality, I have several categories of common and corresponding specific factors: 1st common has 4 specific, and 2nd common has 8 specific, etc.) There are other properties of $y$'s which do not depend on the factors but which are heterogeneous across $y$'s. So, that's why $m>n$.

I need to simulate the distribution of y to estimate its quantile, so I was looking to implement importance sampling to reduce variance of the estimate. Initialy, I was thinking of applying "exponential twisting" change of measure to y (which in this case amounts to changing the mean from $\mathbf{0}$ to $\mathbf{c}$) but this requires taking an inverse of the covariance matrix, which can't be done in this case.

I then thought that maybe I can apply the twisting to x instead by changing their mean from $\mathbf{0}$ to $\mathbf{b} = (\mathbf{A}^T\mathbf{A})^{-1}\mathbf{A}^T\mathbf{c}$ which is a solution to $\mathbf{c} = \mathbf{Ab}$. However, I now have another problem that $\mathbf{A}$ is not even full column rank, i.e. $rank(\mathbf{A})<n$, so it doesn't have even the left inverse and I can't calculate $\mathbf{b}$ as above.. This is because the columns of sensitivities to the specific factors can be combined linearly to give the column of sensitivities to the common factor. For example, say $m=6$ and $n=5$ factors: 1st common without any specific, and 2nd common with 3 specific; then $\mathbf{A}=$

\begin{matrix} b_{11} & b_{12} & a_{12} & 0 & 0\\ b_{21} & b_{12} & a_{12} & 0 & 0\\ b_{31} & b_{22} & 0 & a_{22} & 0\\ b_{41} & b_{22} & 0 & a_{22} & 0\\ b_{51} & b_{32} & 0 & 0 & a_{32}\\ b_{61} & b_{32} & 0 & 0 & a_{32} \end{matrix}

where $b_{ij}$ and $a_{ij}$ are sensitivities to the $j$-th common and specific factors of $y$ belonging to category $i$ of $j$.

Given a vector x of $n$ independent standard normal RVs, and an $m\times n$ matrix A, I was expecting that the linear transformation y=Ax would result in a multivariate normal with a covariance matrix AA$^T$. However, with $m>n$ the resulting $m \times m$ matrix AA$^T$ is not of full rank (if I understand it correctly) and hence is not invertible.

This seems to indicate that there is some "redundancy" in the system of equations y=Ax and that the same amount of information could, somehow, be represented by $n \times n$ matrix A$^*$ instead. Is this correct? And if yes, how does one construct this reduced matrix A$^*$ and then link it with the original system?

I was thinking of maybe performing a principal component analysis on AA$^T$ and then using the column vectors (eigenvectors) with non-zero eigenvalues, but this would still be an $m \times k$ matrix where $k$ is the number of nonzero eigenvalues, so that doesn't seem to be a correct approach. Or maybe I am not applying it in a right way. Should I be taking a matrix made from "non-zero" eigenvalues and applying it to A instead?

Add 1

The background is the following. I have a system which is driven by, say, 4 factors: 1 common and 3 specific. That means that all $y$'s depend on the same common factor, but only those $y$'s that belong to group $i$ depend on the $i$-th specific factor. So, all $y$'s are correlated via the common risk factor and additionally $y$'s belonging to category $i$ are further correlated between them via the $i$-th specific factor. (In reality, I have several categories of common and corresponding specific factors: 1st common has 4 specific, and 2nd common has 8 specific, etc.) There are other properties of $y$'s which do not depend on the factors but which are heterogeneous across $y$'s. So, that's why $m>n$.

I need to simulate the distribution of y to estimate its quantile, so I was looking to implement importance sampling to reduce variance of the estimate. Initialy, I was thinking of applying "exponential twisting" change of measure to y (which in this case amounts to changing the mean from $\mathbf{0}$ to $\mathbf{c}$) but this requires taking an inverse of the covariance matrix, which can't be done in this case.

I then thought that maybe I can apply the twisting to x instead by changing their mean from $\mathbf{0}$ to $\mathbf{b} = (\mathbf{A}^T\mathbf{A})^{-1}\mathbf{A}^T\mathbf{c}$ which is a solution to $\mathbf{c} = \mathbf{Ab}$. However, I now have another problem that $\mathbf{A}$ is not even full column rank, i.e. $rank(\mathbf{A})<n$, so it doesn't have even the left inverse and I can't calculate $\mathbf{b}$ as above.. This is because the columns of sensitivities to the specific factors can be combined linearly to give the column of sensitivities to the common factor. For example, say $m=6$ and $n=5$ factors: 1st common without any specific, and 2nd common with 3 specific; then $\mathbf{A}=$

\begin{matrix} b_{11} & b_{12} & a_{12} & 0 & 0\\ b_{21} & b_{12} & a_{12} & 0 & 0\\ b_{31} & b_{22} & 0 & a_{22} & 0\\ b_{41} & b_{22} & 0 & a_{22} & 0\\ b_{51} & b_{32} & 0 & 0 & a_{32}\\ b_{61} & b_{32} & 0 & 0 & a_{32} \end{matrix}

where $b_{ij}$ and $a_{ij}$ are sensitivities to the $j$-th common and specific factors of $y$ belonging to category $i$ of $j$.

So, the covariance between between $y$'s in the sage category of the 2nd common factor are, for example, $Cov(y_{1},y_{2}) = b_{11}b_{21} + b_{12}^2 + a_{12}^2$, while for those from different categories we have $Cov(y_{1},y_{3}) = b_{11}b_{31} + b_{12}b_{22}$

Given a vector x of $n$ independent standard normal RVs, and an $m\times n$ matrix A, I was expecting that the linear transformation y=Ax would result in a multivariate normal with a covariance matrix AA$^T$. However, with $m>n$ the resulting $m \times m$ matrix AA$^T$ is not of full rank (if I understand it correctly) and hence is not invertible.

This seems to indicate that there is some "redundancy" in the system of equations y=Ax and that the same amount of information could, somehow, be represented by $n \times n$ matrix A$^*$ instead. Is this correct? And if yes, how does one construct this reduced matrix A$^*$ and then link it with the original system?

I was thinking of maybe performing a principal component analysis on AA$^T$ and then using the column vectors (eigenvectors) with non-zero eigenvalues, but this would still be an $m \times k$ matrix where $k$ is the number of nonzero eigenvalues, so that doesn't seem to be a correct approach. Or maybe I am not applying it in a right way. Should I be taking a matrix made from "non-zero" eigenvalues and applying it to A instead?

Add 1

The background is the following. I have a system which is driven by, say, 4 factors: 1 common and 3 specific. That means that all $y$'s depend on the same common factor, but only those $y$'s that belong to group $i$ depend on the $i$-th specific factor. So, all $y$'s are correlated via the common risk factor and additionally $y$'s belonging to category $i$ are further correlated between them via the $i$-th specific factor. (In reality, I have several categories of common and corresponding specific factors: 1st common has 4 specific, and 2nd common has 8 specific, etc.) There are other properties of $y$'s which do not depend on the factors but which are heterogeneous across $y$'s. So, that's why $m>n$.

I need to simulate the distribution of y to estimate its quantile, so I was looking to implement importance sampling to reduce variance of the estimate. Initialy, I was thinking of applying "exponential twisting" change of measure to y (which in this case amounts to changing the mean from $\mathbf{0}$ to $\mathbf{c}$) but this requires taking an inverse of the covariance matrix, which can't be done in this case.

I then thought that maybe I can apply the twisting to x instead by changing their mean from $\mathbf{0}$ to $\mathbf{b} = (\mathbf{A}^T\mathbf{A})^{-1}\mathbf{A}^T\mathbf{c}$ which is a solution to $\mathbf{c} = \mathbf{Ab}$. However, I now have another problem that $\mathbf{A}$ is not even full column rank, i.e. $rank(\mathbf{A})<n$, so it doesn't have even the left inverse and I can't calculate $\mathbf{b}$ as above.. This is because the columns of sensitivities to the specific factors can be combined linearly to give the column of sensitivities to the common factor. For example, say $m=6$, then $\mathbf{A}=$

\begin{matrix} b_{11} & b_{12} & a_{12} & 0 & 0\\ b_{21} & b_{12} & a_{12} & 0 & 0\\ b_{31} & b_{22} & 0 & a_{22} & 0\\ b_{41} & b_{22} & 0 & a_{22} & 0\\ b_{51} & b_{32} & 0 & 0 & a_{32}\\ b_{61} & b_{32} & 0 & 0 & a_{32} \end{matrix}

where $b_{ij}$ and $a_{ij}$ are sensitivities to the $j$-th common and specific factors of $y$ belonging to category $i$ of $j$.

Given a vector x of $n$ independent standard normal RVs, and an $m\times n$ matrix A, I was expecting that the linear transformation y=Ax would result in a multivariate normal with a covariance matrix AA$^T$. However, with $m>n$ the resulting $m \times m$ matrix AA$^T$ is not of full rank (if I understand it correctly) and hence is not invertible.

This seems to indicate that there is some "redundancy" in the system of equations y=Ax and that the same amount of information could, somehow, be represented by $n \times n$ matrix A$^*$ instead. Is this correct? And if yes, how does one construct this reduced matrix A$^*$ and then link it with the original system?

I was thinking of maybe performing a principal component analysis on AA$^T$ and then using the column vectors (eigenvectors) with non-zero eigenvalues, but this would still be an $m \times k$ matrix where $k$ is the number of nonzero eigenvalues, so that doesn't seem to be a correct approach. Or maybe I am not applying it in a right way. Should I be taking a matrix made from "non-zero" eigenvalues and applying it to A instead?

Add 1

The background is the following. I have a system which is driven by, say, 4 factors: 1 common and 3 specific. That means that all $y$'s depend on the same common factor, but only those $y$'s that belong to group $i$ depend on the $i$-th specific factor. So, all $y$'s are correlated via the common risk factor and additionally $y$'s belonging to category $i$ are further correlated between them via the $i$-th specific factor. (In reality, I have several categories of common and corresponding specific factors: 1st common has 4 specific, and 2nd common has 8 specific, etc.) There are other properties of $y$'s which do not depend on the factors but which are heterogeneous across $y$'s. So, that's why $m>n$.

I need to simulate the distribution of y to estimate its quantile, so I was looking to implement importance sampling to reduce variance of the estimate. Initialy, I was thinking of applying "exponential twisting" change of measure to y (which in this case amounts to changing the mean from $\mathbf{0}$ to $\mathbf{c}$) but this requires taking an inverse of the covariance matrix, which can't be done in this case.

I then thought that maybe I can apply the twisting to x instead by changing their mean from $\mathbf{0}$ to $\mathbf{b} = (\mathbf{A}^T\mathbf{A})^{-1}\mathbf{A}^T\mathbf{c}$ which is a solution to $\mathbf{c} = \mathbf{Ab}$. However, I now have another problem that $\mathbf{A}$ is not even full column rank, i.e. $rank(\mathbf{A})<n$, so it doesn't have even the left inverse and I can't calculate $\mathbf{b}$ as above.. This is because the columns of sensitivities to the specific factors can be combined linearly to give the column of sensitivities to the common factor. For example, say $m=6$ and $n=5$ factors: 1st common without any specific, and 2nd common with 3 specific; then $\mathbf{A}=$

\begin{matrix} b_{11} & b_{12} & a_{12} & 0 & 0\\ b_{21} & b_{12} & a_{12} & 0 & 0\\ b_{31} & b_{22} & 0 & a_{22} & 0\\ b_{41} & b_{22} & 0 & a_{22} & 0\\ b_{51} & b_{32} & 0 & 0 & a_{32}\\ b_{61} & b_{32} & 0 & 0 & a_{32} \end{matrix}

where $b_{ij}$ and $a_{ij}$ are sensitivities to the $j$-th common and specific factors of $y$ belonging to category $i$ of $j$.