4
$\begingroup$

I cannot verify the following theorem. Maybe I am doing something wrong, but I don't know what?! Additionally, I'm not sure about the meaning of a constant matrix in the theorem.

Theorem: cov(AX)=Acov(X)A' if A is a constant matrix. (A' means transpose of A, i.e., A^T)

Example:

A (10x5):

    0.8727    0.7145    0.9147    0.6370    0.9760
    0.8241    0.6210    0.0735    0.7318    0.1231
    0.8609    0.7410    0.9865    0.6090    0.7693
    0.1135    0.9563    0.2369    0.5369    0.9876
    0.7349    0.4313    0.9650    0.1657    0.4418
    0.7872    0.6589    0.4490    0.5974    0.7669
    0.4247    0.6411    0.9335    0.5238    0.9153
    0.2126    0.6212    0.6423    0.3651    0.9752
    0.5092    0.6179    0.3732    0.9569    0.1673
    0.8009    0.4691    0.4420    0.6289    0.3132

X (5x5):

    0.6635    0.6323    0.5883    0.9362    0.0998
    0.8298    0.3788    0.4760    0.8170    0.3296
    0.8553    0.6836    0.4701    0.5723    0.6803
    0.0065    0.8085    0.1542    0.5780    0.7653
    0.6389    0.9297    0.1418    0.0438    0.0886

AX (10x5):

    2.5820    2.8701    1.5202    2.3352    1.5188
    1.2083    1.5127    0.9452    1.7493    0.9079
    2.5253    2.7069    1.5259    2.3616    1.5355
    1.7059    1.9482    0.8562    1.3768    0.9861
    1.9542    1.8325    1.1795    1.7078    1.0380
    1.9469    2.2502    1.1887    1.9111    1.1263
    2.2004    2.4239    1.2045    1.7985    1.3707
    1.8312    2.0105    0.9173    1.3278    1.0287
    1.2828    1.7403    0.9404    1.7555    1.2554
    1.5029    1.7859    1.0436    1.7633    1.0443

cov(X) 5x5:

    0.1190   -0.0345    0.0457    0.0139   -0.0482
   -0.0345    0.0429   -0.0316   -0.0557    0.0015
    0.0457   -0.0316    0.0419    0.0563   -0.0132
    0.0139   -0.0557    0.0563    0.1175    0.0102
   -0.0482    0.0015   -0.0132    0.0102    0.1009

cov(AX) (5x5):

    0.2227    0.1984    0.0988    0.1011    0.0827
    0.1984    0.1960    0.0916    0.1063    0.0874
    0.0988    0.0916    0.0576    0.0745    0.0455
    0.1011    0.1063    0.0745    0.1144    0.0618
    0.0827    0.0874    0.0455    0.0618    0.0510

but A*cov(X)*A' (10x10):

  0.2195     0.12119     0.20872    0.077985     0.15076     0.16413     0.17298     0.11964     0.14558     0.15641
 0.12119     0.09362     0.12556  -0.0017266     0.10372     0.08966    0.081364    0.034694    0.099912     0.11002
 0.20872     0.12556     0.20256    0.057506     0.15142     0.15574     0.15915     0.10193     0.14583      0.1577
0.077985  -0.0017266    0.057506     0.11155    0.015719    0.059279    0.089875    0.099947    0.026385    0.015011
 0.15076     0.10372     0.15142    0.015719     0.12378     0.11292     0.10434    0.055612     0.10669     0.12511
 0.16413     0.08966     0.15574    0.059279     0.11292     0.12295     0.12914    0.090107     0.10648     0.11599
 0.17298    0.081364     0.15915    0.089875     0.10434     0.12914     0.14732     0.11382     0.11134     0.11072
 0.11964    0.034694     0.10193    0.099947    0.055612    0.090107     0.11382     0.10483    0.061491    0.057124
 0.14558    0.099912     0.14583    0.026385     0.10669     0.10648     0.11134    0.061491     0.12616     0.12169
 0.15641     0.11002      0.1577    0.015011     0.12511     0.11599     0.11072    0.057124     0.12169     0.13273
Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ In the theorem, $X$ is a column vector of random variables while $A$ is a fixed $n\times n$ matrix of constants, that is, it is not random at all. Thus, $AX$ is a column vector whose covariance matrix is, by definition, $$\operatorname{cov}(AX) = A\operatorname{cov}(X,X^T)A^T = A \operatorname{cov}(X)A^T$$ in your notation. $\endgroup$
    – Dilip Sarwate
    Commented Jul 8, 2014 at 14:05
  • $\begingroup$ Excuse me about this simple question. Would you please verify the dimensions of both sides for me. $\endgroup$
    – remo
    Commented Jul 8, 2014 at 14:35
  • 1
    $\begingroup$ $AX$ is a $n\times 1$ matrix since it is the product of $A$, a $n\times n$ matrix, and $X$, a $n\times 1$ matrix (a.k.a. column vector). The covariance matrix of a column vector is a $n\times n$ matrix whose entries are $\operatorname{cov}(X_i,X_j)$. Thus, your $A \operatorname{cov}(X)A^T$ is the product of three $n\times n$ matrices and is thus also a $n\times n$ matrix. $\endgroup$
    – Dilip Sarwate
    Commented Jul 8, 2014 at 14:50
  • $\begingroup$ What does cov(AX) represent in this case? Is it correct to say it represents the variance of X? $\endgroup$
    – Alex F
    Commented Sep 1, 2017 at 12:49

1 Answer 1

Reset to default
17
$\begingroup$

Let $x \in \mathbb{R}^{p \times 1}$ be a random (column) vector with covariance matrix $\Sigma \in \mathbb{R}^{p \times p}$. Let $y=Ax$, where $A \in \mathbb{R}^{n \times p}$ and $y \in \mathbb{R}^{n \times 1}$. The theorem says that $$ \mathrm{Cov}(Ax) = A \Sigma A^T $$ The proof of the theorem is straightforward: $$ \mathrm{Cov}(y) = \mathrm{Cov}(Ax) = \mathbb{E}[(Ax - \mathbb{E}[Ax])(Ax - \mathbb{E}[Ax])^T] \\ = \mathbb{E}[(Ax - A \mathbb{E}[x])(Ax - A\mathbb{E}[x])^T] \\ = \mathbb{E}[A (x - \mathbb{E}[x])(x - \mathbb{E}[x])^T A^T] \\ = A \underbrace{\mathbb{E}[(x - \mathbb{E}[x])(x - \mathbb{E}[x])^T]}_{=\Sigma} A^T \\ $$ and this concludes the proof.

EDIT: Suppose now that $Y = AX$, where $X \in \mathbb{R}^{p \times p}$, $A \in \mathbb{R}^{n \times p}$ and $Y \in \mathbb{R}^{n \times p}$. Essentially, everything could be written in standard notation by using vectorization, i.e., defining $$ x \triangleq \mathrm{vec}(X), \quad y \triangleq \mathrm{vec}(Y) $$ with $x \in \mathbb{R}^{p^2 \times 1}$, $y \in \mathbb{R}^{np \times 1}$ and writing the problem in the form $y = Hx$, where $H$ is given by $$ H = \begin{bmatrix} A \\ & A \\ & & \ddots \\ & & & A \end{bmatrix} \in \mathbb{R}^{np \times p^2} $$ After that, the theorem can be applied and the proof remains practically unchanged.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Why X must be symmetric. I know cov(X) must be symmetric, but what about X, itself? $\endgroup$
    – remo
    Commented Jul 8, 2014 at 14:33
  • $\begingroup$ In your code, you were using "X" as a covariance matrix (look at the first comment). What I meant was: the covariance matrix must be symmetric. $\endgroup$
    – PseudoRandom
    Commented Jul 8, 2014 at 14:35
  • $\begingroup$ I cannot find such a comment, By the way I added cov(X) to my question and I mean X is a matrix of variables not necessarily symmetric. If there is such a formulation for cov(AX)? (Please see X as a matrix and not a random variable) $\endgroup$
    – remo
    Commented Jul 8, 2014 at 14:42
  • $\begingroup$ Ok, I got what you are saying. If $X$ is a matrix of random variables, you must beforehand use the "vec" operator and convert it into a column vector and adjust the model accordingly. After that, you can apply the theorem. The actual theorem is about the covariance of $Ax$ (a matrix and a column vector). If you want a formulation for $AX$, it could probably be derived, but using the "vec" operator is simpler. $\endgroup$
    – PseudoRandom
    Commented Jul 8, 2014 at 14:45
  • $\begingroup$ I have substantially edited my answer. Tell me if it answers your question or if there is something more you would like to see. $\endgroup$
    – PseudoRandom
    Commented Jul 8, 2014 at 15:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.