3
$\begingroup$

I cannot verify the following theorem. Maybe I am doing something wrong, but I don't know what?! Additionally, I'm not sure about the meaning of a constant matrix in the theorem.

Theorem: cov(AX)=Acov(X)A' if A is a constant matrix. (A' means transpose of A, i.e., A^T)

Example:

A (10x5):

    0.8727    0.7145    0.9147    0.6370    0.9760
    0.8241    0.6210    0.0735    0.7318    0.1231
    0.8609    0.7410    0.9865    0.6090    0.7693
    0.1135    0.9563    0.2369    0.5369    0.9876
    0.7349    0.4313    0.9650    0.1657    0.4418
    0.7872    0.6589    0.4490    0.5974    0.7669
    0.4247    0.6411    0.9335    0.5238    0.9153
    0.2126    0.6212    0.6423    0.3651    0.9752
    0.5092    0.6179    0.3732    0.9569    0.1673
    0.8009    0.4691    0.4420    0.6289    0.3132

X (5x5):

    0.6635    0.6323    0.5883    0.9362    0.0998
    0.8298    0.3788    0.4760    0.8170    0.3296
    0.8553    0.6836    0.4701    0.5723    0.6803
    0.0065    0.8085    0.1542    0.5780    0.7653
    0.6389    0.9297    0.1418    0.0438    0.0886

AX (10x5):

    2.5820    2.8701    1.5202    2.3352    1.5188
    1.2083    1.5127    0.9452    1.7493    0.9079
    2.5253    2.7069    1.5259    2.3616    1.5355
    1.7059    1.9482    0.8562    1.3768    0.9861
    1.9542    1.8325    1.1795    1.7078    1.0380
    1.9469    2.2502    1.1887    1.9111    1.1263
    2.2004    2.4239    1.2045    1.7985    1.3707
    1.8312    2.0105    0.9173    1.3278    1.0287
    1.2828    1.7403    0.9404    1.7555    1.2554
    1.5029    1.7859    1.0436    1.7633    1.0443

cov(X) 5x5:

    0.1190   -0.0345    0.0457    0.0139   -0.0482
   -0.0345    0.0429   -0.0316   -0.0557    0.0015
    0.0457   -0.0316    0.0419    0.0563   -0.0132
    0.0139   -0.0557    0.0563    0.1175    0.0102
   -0.0482    0.0015   -0.0132    0.0102    0.1009

cov(AX) (5x5):

    0.2227    0.1984    0.0988    0.1011    0.0827
    0.1984    0.1960    0.0916    0.1063    0.0874
    0.0988    0.0916    0.0576    0.0745    0.0455
    0.1011    0.1063    0.0745    0.1144    0.0618
    0.0827    0.0874    0.0455    0.0618    0.0510

but A*cov(X)*A' (10x10):

  0.2195     0.12119     0.20872    0.077985     0.15076     0.16413     0.17298     0.11964     0.14558     0.15641
 0.12119     0.09362     0.12556  -0.0017266     0.10372     0.08966    0.081364    0.034694    0.099912     0.11002
 0.20872     0.12556     0.20256    0.057506     0.15142     0.15574     0.15915     0.10193     0.14583      0.1577
0.077985  -0.0017266    0.057506     0.11155    0.015719    0.059279    0.089875    0.099947    0.026385    0.015011
 0.15076     0.10372     0.15142    0.015719     0.12378     0.11292     0.10434    0.055612     0.10669     0.12511
 0.16413     0.08966     0.15574    0.059279     0.11292     0.12295     0.12914    0.090107     0.10648     0.11599
 0.17298    0.081364     0.15915    0.089875     0.10434     0.12914     0.14732     0.11382     0.11134     0.11072
 0.11964    0.034694     0.10193    0.099947    0.055612    0.090107     0.11382     0.10483    0.061491    0.057124
 0.14558    0.099912     0.14583    0.026385     0.10669     0.10648     0.11134    0.061491     0.12616     0.12169
 0.15641     0.11002      0.1577    0.015011     0.12511     0.11599     0.11072    0.057124     0.12169     0.13273
$\endgroup$
  • 2
    $\begingroup$ In the theorem, $X$ is a column vector of random variables while $A$ is a fixed $n\times n$ matrix of constants, that is, it is not random at all. Thus, $AX$ is a column vector whose covariance matrix is, by definition, $$\operatorname{cov}(AX) = A\operatorname{cov}(X,X^T)A^T = A \operatorname{cov}(X)A^T$$ in your notation. $\endgroup$ – Dilip Sarwate Jul 8 '14 at 14:05
  • $\begingroup$ Excuse me about this simple question. Would you please verify the dimensions of both sides for me. $\endgroup$ – remo Jul 8 '14 at 14:35
  • 1
    $\begingroup$ $AX$ is a $n\times 1$ matrix since it is the product of $A$, a $n\times n$ matrix, and $X$, a $n\times 1$ matrix (a.k.a. column vector). The covariance matrix of a column vector is a $n\times n$ matrix whose entries are $\operatorname{cov}(X_i,X_j)$. Thus, your $A \operatorname{cov}(X)A^T$ is the product of three $n\times n$ matrices and is thus also a $n\times n$ matrix. $\endgroup$ – Dilip Sarwate Jul 8 '14 at 14:50
  • $\begingroup$ What does cov(AX) represent in this case? Is it correct to say it represents the variance of X? $\endgroup$ – Alex F Sep 1 '17 at 12:49
12
$\begingroup$

Let $x \in \mathbb{R}^{p \times 1}$ be a random (column) vector with covariance matrix $\Sigma \in \mathbb{R}^{p \times p}$. Let $y=Ax$, where $A \in \mathbb{R}^{n \times p}$ and $y \in \mathbb{R}^{n \times 1}$. The theorem says that $$ \mathrm{Cov}(Ax) = A \Sigma A^T $$ The proof of the theorem is straightforward: $$ \mathrm{Cov}(y) = \mathrm{Cov}(Ax) = \mathbb{E}[(Ax - \mathbb{E}[Ax])(Ax - \mathbb{E}[Ax])^T] \\ = \mathbb{E}[(Ax - A \mathbb{E}[x])(Ax - A\mathbb{E}[x])^T] \\ = \mathbb{E}[A (x - \mathbb{E}[x])(x - \mathbb{E}[x])^T A^T] \\ = A \underbrace{\mathbb{E}[(x - \mathbb{E}[x])(x - \mathbb{E}[x])^T]}_{=\Sigma} A^T \\ $$ and this concludes the proof.


EDIT: Suppose now that $Y = AX$, where $X \in \mathbb{R}^{p \times p}$, $A \in \mathbb{R}^{n \times p}$ and $Y \in \mathbb{R}^{n \times p}$. Essentially, everything could be written in standard notation by using vectorization, i.e., defining $$ x \triangleq \mathrm{vec}(X), \quad y \triangleq \mathrm{vec}(Y) $$ with $x \in \mathbb{R}^{p^2 \times 1}$, $y \in \mathbb{R}^{np \times 1}$ and writing the problem in the form $y = Hx$, where $H$ is given by $$ H = \begin{bmatrix} A \\ & A \\ & & \ddots \\ & & & A \end{bmatrix} \in \mathbb{R}^{np \times p^2} $$ After that, the theorem can be applied and the proof remains practically unchanged.

$\endgroup$
  • $\begingroup$ Why X must be symmetric. I know cov(X) must be symmetric, but what about X, itself? $\endgroup$ – remo Jul 8 '14 at 14:33
  • $\begingroup$ In your code, you were using "X" as a covariance matrix (look at the first comment). What I meant was: the covariance matrix must be symmetric. $\endgroup$ – PseudoRandom Jul 8 '14 at 14:35
  • $\begingroup$ I cannot find such a comment, By the way I added cov(X) to my question and I mean X is a matrix of variables not necessarily symmetric. If there is such a formulation for cov(AX)? (Please see X as a matrix and not a random variable) $\endgroup$ – remo Jul 8 '14 at 14:42
  • $\begingroup$ Ok, I got what you are saying. If $X$ is a matrix of random variables, you must beforehand use the "vec" operator and convert it into a column vector and adjust the model accordingly. After that, you can apply the theorem. The actual theorem is about the covariance of $Ax$ (a matrix and a column vector). If you want a formulation for $AX$, it could probably be derived, but using the "vec" operator is simpler. $\endgroup$ – PseudoRandom Jul 8 '14 at 14:45
  • $\begingroup$ I have substantially edited my answer. Tell me if it answers your question or if there is something more you would like to see. $\endgroup$ – PseudoRandom Jul 8 '14 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.